13ff01b23SMartin Matuska.\" 23ff01b23SMartin Matuska.\" Copyright (c) 2013 by Turbo Fredriksson <turbo@bayour.com>. All rights reserved. 33ff01b23SMartin Matuska.\" Copyright (c) 2019, 2021 by Delphix. All rights reserved. 43ff01b23SMartin Matuska.\" Copyright (c) 2019 Datto Inc. 53ff01b23SMartin Matuska.\" The contents of this file are subject to the terms of the Common Development 63ff01b23SMartin Matuska.\" and Distribution License (the "License"). You may not use this file except 73ff01b23SMartin Matuska.\" in compliance with the License. You can obtain a copy of the license at 8271171e0SMartin Matuska.\" usr/src/OPENSOLARIS.LICENSE or https://opensource.org/licenses/CDDL-1.0. 93ff01b23SMartin Matuska.\" 103ff01b23SMartin Matuska.\" See the License for the specific language governing permissions and 113ff01b23SMartin Matuska.\" limitations under the License. When distributing Covered Code, include this 123ff01b23SMartin Matuska.\" CDDL HEADER in each file and include the License file at 133ff01b23SMartin Matuska.\" usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this 143ff01b23SMartin Matuska.\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your 153ff01b23SMartin Matuska.\" own identifying information: 163ff01b23SMartin Matuska.\" Portions Copyright [yyyy] [name of copyright owner] 173ff01b23SMartin Matuska.\" 18*15f0b8c3SMartin Matuska.Dd January 10, 2023 193ff01b23SMartin Matuska.Dt ZFS 4 203ff01b23SMartin Matuska.Os 213ff01b23SMartin Matuska. 223ff01b23SMartin Matuska.Sh NAME 233ff01b23SMartin Matuska.Nm zfs 243ff01b23SMartin Matuska.Nd tuning of the ZFS kernel module 253ff01b23SMartin Matuska. 263ff01b23SMartin Matuska.Sh DESCRIPTION 273ff01b23SMartin MatuskaThe ZFS module supports these parameters: 283ff01b23SMartin Matuska.Bl -tag -width Ds 29dbd5678dSMartin Matuska.It Sy dbuf_cache_max_bytes Ns = Ns Sy UINT64_MAX Ns B Pq u64 303ff01b23SMartin MatuskaMaximum size in bytes of the dbuf cache. 313ff01b23SMartin MatuskaThe target size is determined by the MIN versus 323ff01b23SMartin Matuska.No 1/2^ Ns Sy dbuf_cache_shift Pq 1/32nd 333ff01b23SMartin Matuskaof the target ARC size. 343ff01b23SMartin MatuskaThe behavior of the dbuf cache and its associated settings 353ff01b23SMartin Matuskacan be observed via the 363ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs/dbufstats 373ff01b23SMartin Matuskakstat. 383ff01b23SMartin Matuska. 39dbd5678dSMartin Matuska.It Sy dbuf_metadata_cache_max_bytes Ns = Ns Sy UINT64_MAX Ns B Pq u64 403ff01b23SMartin MatuskaMaximum size in bytes of the metadata dbuf cache. 413ff01b23SMartin MatuskaThe target size is determined by the MIN versus 423ff01b23SMartin Matuska.No 1/2^ Ns Sy dbuf_metadata_cache_shift Pq 1/64th 433ff01b23SMartin Matuskaof the target ARC size. 443ff01b23SMartin MatuskaThe behavior of the metadata dbuf cache and its associated settings 453ff01b23SMartin Matuskacan be observed via the 463ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs/dbufstats 473ff01b23SMartin Matuskakstat. 483ff01b23SMartin Matuska. 493ff01b23SMartin Matuska.It Sy dbuf_cache_hiwater_pct Ns = Ns Sy 10 Ns % Pq uint 503ff01b23SMartin MatuskaThe percentage over 513ff01b23SMartin Matuska.Sy dbuf_cache_max_bytes 523ff01b23SMartin Matuskawhen dbufs must be evicted directly. 533ff01b23SMartin Matuska. 543ff01b23SMartin Matuska.It Sy dbuf_cache_lowater_pct Ns = Ns Sy 10 Ns % Pq uint 553ff01b23SMartin MatuskaThe percentage below 563ff01b23SMartin Matuska.Sy dbuf_cache_max_bytes 573ff01b23SMartin Matuskawhen the evict thread stops evicting dbufs. 583ff01b23SMartin Matuska. 59be181ee2SMartin Matuska.It Sy dbuf_cache_shift Ns = Ns Sy 5 Pq uint 603ff01b23SMartin MatuskaSet the size of the dbuf cache 613ff01b23SMartin Matuska.Pq Sy dbuf_cache_max_bytes 623ff01b23SMartin Matuskato a log2 fraction of the target ARC size. 633ff01b23SMartin Matuska. 64be181ee2SMartin Matuska.It Sy dbuf_metadata_cache_shift Ns = Ns Sy 6 Pq uint 653ff01b23SMartin MatuskaSet the size of the dbuf metadata cache 663ff01b23SMartin Matuska.Pq Sy dbuf_metadata_cache_max_bytes 673ff01b23SMartin Matuskato a log2 fraction of the target ARC size. 683ff01b23SMartin Matuska. 69be181ee2SMartin Matuska.It Sy dbuf_mutex_cache_shift Ns = Ns Sy 0 Pq uint 70be181ee2SMartin MatuskaSet the size of the mutex array for the dbuf cache. 71be181ee2SMartin MatuskaWhen set to 72be181ee2SMartin Matuska.Sy 0 73be181ee2SMartin Matuskathe array is dynamically sized based on total system memory. 74be181ee2SMartin Matuska. 75be181ee2SMartin Matuska.It Sy dmu_object_alloc_chunk_shift Ns = Ns Sy 7 Po 128 Pc Pq uint 763ff01b23SMartin Matuskadnode slots allocated in a single operation as a power of 2. 773ff01b23SMartin MatuskaThe default value minimizes lock contention for the bulk operation performed. 783ff01b23SMartin Matuska. 79be181ee2SMartin Matuska.It Sy dmu_prefetch_max Ns = Ns Sy 134217728 Ns B Po 128 MiB Pc Pq uint 803ff01b23SMartin MatuskaLimit the amount we can prefetch with one call to this amount in bytes. 813ff01b23SMartin MatuskaThis helps to limit the amount of memory that can be used by prefetching. 823ff01b23SMartin Matuska. 833ff01b23SMartin Matuska.It Sy ignore_hole_birth Pq int 843ff01b23SMartin MatuskaAlias for 853ff01b23SMartin Matuska.Sy send_holes_without_birth_time . 863ff01b23SMartin Matuska. 873ff01b23SMartin Matuska.It Sy l2arc_feed_again Ns = Ns Sy 1 Ns | Ns 0 Pq int 883ff01b23SMartin MatuskaTurbo L2ARC warm-up. 893ff01b23SMartin MatuskaWhen the L2ARC is cold the fill interval will be set as fast as possible. 903ff01b23SMartin Matuska. 91dbd5678dSMartin Matuska.It Sy l2arc_feed_min_ms Ns = Ns Sy 200 Pq u64 923ff01b23SMartin MatuskaMin feed interval in milliseconds. 933ff01b23SMartin MatuskaRequires 943ff01b23SMartin Matuska.Sy l2arc_feed_again Ns = Ns Ar 1 953ff01b23SMartin Matuskaand only applicable in related situations. 963ff01b23SMartin Matuska. 97dbd5678dSMartin Matuska.It Sy l2arc_feed_secs Ns = Ns Sy 1 Pq u64 983ff01b23SMartin MatuskaSeconds between L2ARC writing. 993ff01b23SMartin Matuska. 100dbd5678dSMartin Matuska.It Sy l2arc_headroom Ns = Ns Sy 2 Pq u64 1013ff01b23SMartin MatuskaHow far through the ARC lists to search for L2ARC cacheable content, 1023ff01b23SMartin Matuskaexpressed as a multiplier of 1033ff01b23SMartin Matuska.Sy l2arc_write_max . 1043ff01b23SMartin MatuskaARC persistence across reboots can be achieved with persistent L2ARC 1053ff01b23SMartin Matuskaby setting this parameter to 1063ff01b23SMartin Matuska.Sy 0 , 1073ff01b23SMartin Matuskaallowing the full length of ARC lists to be searched for cacheable content. 1083ff01b23SMartin Matuska. 109dbd5678dSMartin Matuska.It Sy l2arc_headroom_boost Ns = Ns Sy 200 Ns % Pq u64 1103ff01b23SMartin MatuskaScales 1113ff01b23SMartin Matuska.Sy l2arc_headroom 1123ff01b23SMartin Matuskaby this percentage when L2ARC contents are being successfully compressed 1133ff01b23SMartin Matuskabefore writing. 1143ff01b23SMartin MatuskaA value of 1153ff01b23SMartin Matuska.Sy 100 1163ff01b23SMartin Matuskadisables this feature. 1173ff01b23SMartin Matuska. 118dae17134SMartin Matuska.It Sy l2arc_exclude_special Ns = Ns Sy 0 Ns | Ns 1 Pq int 119e92ffd9bSMartin MatuskaControls whether buffers present on special vdevs are eligible for caching 120dae17134SMartin Matuskainto L2ARC. 121dae17134SMartin MatuskaIf set to 1, exclude dbufs on special vdevs from being cached to L2ARC. 122dae17134SMartin Matuska. 1233ff01b23SMartin Matuska.It Sy l2arc_mfuonly Ns = Ns Sy 0 Ns | Ns 1 Pq int 1243ff01b23SMartin MatuskaControls whether only MFU metadata and data are cached from ARC into L2ARC. 1253ff01b23SMartin MatuskaThis may be desired to avoid wasting space on L2ARC when reading/writing large 1263ff01b23SMartin Matuskaamounts of data that are not expected to be accessed more than once. 1273ff01b23SMartin Matuska.Pp 1283ff01b23SMartin MatuskaThe default is off, 1293ff01b23SMartin Matuskameaning both MRU and MFU data and metadata are cached. 1303ff01b23SMartin MatuskaWhen turning off this feature, some MRU buffers will still be present 1313ff01b23SMartin Matuskain ARC and eventually cached on L2ARC. 1323ff01b23SMartin Matuska.No If Sy l2arc_noprefetch Ns = Ns Sy 0 , 1333ff01b23SMartin Matuskasome prefetched buffers will be cached to L2ARC, and those might later 1343ff01b23SMartin Matuskatransition to MRU, in which case the 1353ff01b23SMartin Matuska.Sy l2arc_mru_asize No arcstat will not be Sy 0 . 1363ff01b23SMartin Matuska.Pp 1373ff01b23SMartin MatuskaRegardless of 1383ff01b23SMartin Matuska.Sy l2arc_noprefetch , 1393ff01b23SMartin Matuskasome MFU buffers might be evicted from ARC, 1403ff01b23SMartin Matuskaaccessed later on as prefetches and transition to MRU as prefetches. 1413ff01b23SMartin MatuskaIf accessed again they are counted as MRU and the 1423ff01b23SMartin Matuska.Sy l2arc_mru_asize No arcstat will not be Sy 0 . 1433ff01b23SMartin Matuska.Pp 1443ff01b23SMartin MatuskaThe ARC status of L2ARC buffers when they were first cached in 1453ff01b23SMartin MatuskaL2ARC can be seen in the 1463ff01b23SMartin Matuska.Sy l2arc_mru_asize , Sy l2arc_mfu_asize , No and Sy l2arc_prefetch_asize 1473ff01b23SMartin Matuskaarcstats when importing the pool or onlining a cache 1483ff01b23SMartin Matuskadevice if persistent L2ARC is enabled. 1493ff01b23SMartin Matuska.Pp 1503ff01b23SMartin MatuskaThe 1513ff01b23SMartin Matuska.Sy evict_l2_eligible_mru 1523ff01b23SMartin Matuskaarcstat does not take into account if this option is enabled as the information 1533ff01b23SMartin Matuskaprovided by the 1543ff01b23SMartin Matuska.Sy evict_l2_eligible_m[rf]u 1553ff01b23SMartin Matuskaarcstats can be used to decide if toggling this option is appropriate 1563ff01b23SMartin Matuskafor the current workload. 1573ff01b23SMartin Matuska. 158be181ee2SMartin Matuska.It Sy l2arc_meta_percent Ns = Ns Sy 33 Ns % Pq uint 1593ff01b23SMartin MatuskaPercent of ARC size allowed for L2ARC-only headers. 1603ff01b23SMartin MatuskaSince L2ARC buffers are not evicted on memory pressure, 1613ff01b23SMartin Matuskatoo many headers on a system with an irrationally large L2ARC 1623ff01b23SMartin Matuskacan render it slow or unusable. 1633ff01b23SMartin MatuskaThis parameter limits L2ARC writes and rebuilds to achieve the target. 1643ff01b23SMartin Matuska. 165dbd5678dSMartin Matuska.It Sy l2arc_trim_ahead Ns = Ns Sy 0 Ns % Pq u64 1663ff01b23SMartin MatuskaTrims ahead of the current write size 1673ff01b23SMartin Matuska.Pq Sy l2arc_write_max 1683ff01b23SMartin Matuskaon L2ARC devices by this percentage of write size if we have filled the device. 1693ff01b23SMartin MatuskaIf set to 1703ff01b23SMartin Matuska.Sy 100 1713ff01b23SMartin Matuskawe TRIM twice the space required to accommodate upcoming writes. 1723ff01b23SMartin MatuskaA minimum of 173716fd348SMartin Matuska.Sy 64 MiB 1743ff01b23SMartin Matuskawill be trimmed. 1753ff01b23SMartin MatuskaIt also enables TRIM of the whole L2ARC device upon creation 1763ff01b23SMartin Matuskaor addition to an existing pool or if the header of the device is 1773ff01b23SMartin Matuskainvalid upon importing a pool or onlining a cache device. 1783ff01b23SMartin MatuskaA value of 1793ff01b23SMartin Matuska.Sy 0 1803ff01b23SMartin Matuskadisables TRIM on L2ARC altogether and is the default as it can put significant 1813ff01b23SMartin Matuskastress on the underlying storage devices. 1823ff01b23SMartin MatuskaThis will vary depending of how well the specific device handles these commands. 1833ff01b23SMartin Matuska. 1843ff01b23SMartin Matuska.It Sy l2arc_noprefetch Ns = Ns Sy 1 Ns | Ns 0 Pq int 1853ff01b23SMartin MatuskaDo not write buffers to L2ARC if they were prefetched but not used by 1863ff01b23SMartin Matuskaapplications. 1873ff01b23SMartin MatuskaIn case there are prefetched buffers in L2ARC and this option 1883ff01b23SMartin Matuskais later set, we do not read the prefetched buffers from L2ARC. 1893ff01b23SMartin MatuskaUnsetting this option is useful for caching sequential reads from the 1903ff01b23SMartin Matuskadisks to L2ARC and serve those reads from L2ARC later on. 1913ff01b23SMartin MatuskaThis may be beneficial in case the L2ARC device is significantly faster 1923ff01b23SMartin Matuskain sequential reads than the disks of the pool. 1933ff01b23SMartin Matuska.Pp 1943ff01b23SMartin MatuskaUse 1953ff01b23SMartin Matuska.Sy 1 1963ff01b23SMartin Matuskato disable and 1973ff01b23SMartin Matuska.Sy 0 1983ff01b23SMartin Matuskato enable caching/reading prefetches to/from L2ARC. 1993ff01b23SMartin Matuska. 2003ff01b23SMartin Matuska.It Sy l2arc_norw Ns = Ns Sy 0 Ns | Ns 1 Pq int 2013ff01b23SMartin MatuskaNo reads during writes. 2023ff01b23SMartin Matuska. 203dbd5678dSMartin Matuska.It Sy l2arc_write_boost Ns = Ns Sy 8388608 Ns B Po 8 MiB Pc Pq u64 2043ff01b23SMartin MatuskaCold L2ARC devices will have 2053ff01b23SMartin Matuska.Sy l2arc_write_max 2063ff01b23SMartin Matuskaincreased by this amount while they remain cold. 2073ff01b23SMartin Matuska. 208dbd5678dSMartin Matuska.It Sy l2arc_write_max Ns = Ns Sy 8388608 Ns B Po 8 MiB Pc Pq u64 2093ff01b23SMartin MatuskaMax write bytes per interval. 2103ff01b23SMartin Matuska. 2113ff01b23SMartin Matuska.It Sy l2arc_rebuild_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 2123ff01b23SMartin MatuskaRebuild the L2ARC when importing a pool (persistent L2ARC). 2133ff01b23SMartin MatuskaThis can be disabled if there are problems importing a pool 2143ff01b23SMartin Matuskaor attaching an L2ARC device (e.g. the L2ARC device is slow 2153ff01b23SMartin Matuskain reading stored log metadata, or the metadata 2163ff01b23SMartin Matuskahas become somehow fragmented/unusable). 2173ff01b23SMartin Matuska. 218dbd5678dSMartin Matuska.It Sy l2arc_rebuild_blocks_min_l2size Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq u64 2193ff01b23SMartin MatuskaMininum size of an L2ARC device required in order to write log blocks in it. 2203ff01b23SMartin MatuskaThe log blocks are used upon importing the pool to rebuild the persistent L2ARC. 2213ff01b23SMartin Matuska.Pp 222716fd348SMartin MatuskaFor L2ARC devices less than 1 GiB, the amount of data 2233ff01b23SMartin Matuska.Fn l2arc_evict 2243ff01b23SMartin Matuskaevicts is significant compared to the amount of restored L2ARC data. 2253ff01b23SMartin MatuskaIn this case, do not write log blocks in L2ARC in order not to waste space. 2263ff01b23SMartin Matuska. 227dbd5678dSMartin Matuska.It Sy metaslab_aliquot Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 2283ff01b23SMartin MatuskaMetaslab granularity, in bytes. 2293ff01b23SMartin MatuskaThis is roughly similar to what would be referred to as the "stripe size" 2303ff01b23SMartin Matuskain traditional RAID arrays. 231716fd348SMartin MatuskaIn normal operation, ZFS will try to write this amount of data to each disk 232716fd348SMartin Matuskabefore moving on to the next top-level vdev. 2333ff01b23SMartin Matuska. 2343ff01b23SMartin Matuska.It Sy metaslab_bias_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 2353ff01b23SMartin MatuskaEnable metaslab group biasing based on their vdevs' over- or under-utilization 2363ff01b23SMartin Matuskarelative to the pool. 2373ff01b23SMartin Matuska. 238dbd5678dSMartin Matuska.It Sy metaslab_force_ganging Ns = Ns Sy 16777217 Ns B Po 16 MiB + 1 B Pc Pq u64 2393ff01b23SMartin MatuskaMake some blocks above a certain size be gang blocks. 2403ff01b23SMartin MatuskaThis option is used by the test suite to facilitate testing. 2413ff01b23SMartin Matuska. 242*15f0b8c3SMartin Matuska.It Sy zfs_default_bs Ns = Ns Sy 9 Po 512 B Pc Pq int 243*15f0b8c3SMartin MatuskaDefault dnode block size as a power of 2. 244*15f0b8c3SMartin Matuska. 245*15f0b8c3SMartin Matuska.It Sy zfs_default_ibs Ns = Ns Sy 17 Po 128 KiB Pc Pq int 246*15f0b8c3SMartin MatuskaDefault dnode indirect block size as a power of 2. 247*15f0b8c3SMartin Matuska. 248dbd5678dSMartin Matuska.It Sy zfs_history_output_max Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 2493ff01b23SMartin MatuskaWhen attempting to log an output nvlist of an ioctl in the on-disk history, 2503ff01b23SMartin Matuskathe output will not be stored if it is larger than this size (in bytes). 2513ff01b23SMartin MatuskaThis must be less than 252716fd348SMartin Matuska.Sy DMU_MAX_ACCESS Pq 64 MiB . 2533ff01b23SMartin MatuskaThis applies primarily to 2543ff01b23SMartin Matuska.Fn zfs_ioc_channel_program Pq cf. Xr zfs-program 8 . 2553ff01b23SMartin Matuska. 2563ff01b23SMartin Matuska.It Sy zfs_keep_log_spacemaps_at_export Ns = Ns Sy 0 Ns | Ns 1 Pq int 2573ff01b23SMartin MatuskaPrevent log spacemaps from being destroyed during pool exports and destroys. 2583ff01b23SMartin Matuska. 2593ff01b23SMartin Matuska.It Sy zfs_metaslab_segment_weight_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 2603ff01b23SMartin MatuskaEnable/disable segment-based metaslab selection. 2613ff01b23SMartin Matuska. 2623ff01b23SMartin Matuska.It Sy zfs_metaslab_switch_threshold Ns = Ns Sy 2 Pq int 2633ff01b23SMartin MatuskaWhen using segment-based metaslab selection, continue allocating 2643ff01b23SMartin Matuskafrom the active metaslab until this option's 2653ff01b23SMartin Matuskaworth of buckets have been exhausted. 2663ff01b23SMartin Matuska. 2673ff01b23SMartin Matuska.It Sy metaslab_debug_load Ns = Ns Sy 0 Ns | Ns 1 Pq int 2683ff01b23SMartin MatuskaLoad all metaslabs during pool import. 2693ff01b23SMartin Matuska. 2703ff01b23SMartin Matuska.It Sy metaslab_debug_unload Ns = Ns Sy 0 Ns | Ns 1 Pq int 2713ff01b23SMartin MatuskaPrevent metaslabs from being unloaded. 2723ff01b23SMartin Matuska. 2733ff01b23SMartin Matuska.It Sy metaslab_fragmentation_factor_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 2743ff01b23SMartin MatuskaEnable use of the fragmentation metric in computing metaslab weights. 2753ff01b23SMartin Matuska. 276be181ee2SMartin Matuska.It Sy metaslab_df_max_search Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint 2773ff01b23SMartin MatuskaMaximum distance to search forward from the last offset. 2783ff01b23SMartin MatuskaWithout this limit, fragmented pools can see 2793ff01b23SMartin Matuska.Em >100`000 2803ff01b23SMartin Matuskaiterations and 2813ff01b23SMartin Matuska.Fn metaslab_block_picker 2823ff01b23SMartin Matuskabecomes the performance limiting factor on high-performance storage. 2833ff01b23SMartin Matuska.Pp 2843ff01b23SMartin MatuskaWith the default setting of 285716fd348SMartin Matuska.Sy 16 MiB , 2863ff01b23SMartin Matuskawe typically see less than 2873ff01b23SMartin Matuska.Em 500 2883ff01b23SMartin Matuskaiterations, even with very fragmented 2893ff01b23SMartin Matuska.Sy ashift Ns = Ns Sy 9 2903ff01b23SMartin Matuskapools. 2913ff01b23SMartin MatuskaThe maximum number of iterations possible is 2923ff01b23SMartin Matuska.Sy metaslab_df_max_search / 2^(ashift+1) . 2933ff01b23SMartin MatuskaWith the default setting of 294716fd348SMartin Matuska.Sy 16 MiB 2953ff01b23SMartin Matuskathis is 2963ff01b23SMartin Matuska.Em 16*1024 Pq with Sy ashift Ns = Ns Sy 9 2973ff01b23SMartin Matuskaor 2983ff01b23SMartin Matuska.Em 2*1024 Pq with Sy ashift Ns = Ns Sy 12 . 2993ff01b23SMartin Matuska. 3003ff01b23SMartin Matuska.It Sy metaslab_df_use_largest_segment Ns = Ns Sy 0 Ns | Ns 1 Pq int 3013ff01b23SMartin MatuskaIf not searching forward (due to 3023ff01b23SMartin Matuska.Sy metaslab_df_max_search , metaslab_df_free_pct , 3033ff01b23SMartin Matuska.No or Sy metaslab_df_alloc_threshold ) , 3043ff01b23SMartin Matuskathis tunable controls which segment is used. 3053ff01b23SMartin MatuskaIf set, we will use the largest free segment. 3063ff01b23SMartin MatuskaIf unset, we will use a segment of at least the requested size. 3073ff01b23SMartin Matuska. 308dbd5678dSMartin Matuska.It Sy zfs_metaslab_max_size_cache_sec Ns = Ns Sy 3600 Ns s Po 1 hour Pc Pq u64 3093ff01b23SMartin MatuskaWhen we unload a metaslab, we cache the size of the largest free chunk. 3103ff01b23SMartin MatuskaWe use that cached size to determine whether or not to load a metaslab 3113ff01b23SMartin Matuskafor a given allocation. 3123ff01b23SMartin MatuskaAs more frees accumulate in that metaslab while it's unloaded, 3133ff01b23SMartin Matuskathe cached max size becomes less and less accurate. 3143ff01b23SMartin MatuskaAfter a number of seconds controlled by this tunable, 3153ff01b23SMartin Matuskawe stop considering the cached max size and start 3163ff01b23SMartin Matuskaconsidering only the histogram instead. 3173ff01b23SMartin Matuska. 318be181ee2SMartin Matuska.It Sy zfs_metaslab_mem_limit Ns = Ns Sy 25 Ns % Pq uint 3193ff01b23SMartin MatuskaWhen we are loading a new metaslab, we check the amount of memory being used 3203ff01b23SMartin Matuskato store metaslab range trees. 3213ff01b23SMartin MatuskaIf it is over a threshold, we attempt to unload the least recently used metaslab 3223ff01b23SMartin Matuskato prevent the system from clogging all of its memory with range trees. 3233ff01b23SMartin MatuskaThis tunable sets the percentage of total system memory that is the threshold. 3243ff01b23SMartin Matuska. 3253ff01b23SMartin Matuska.It Sy zfs_metaslab_try_hard_before_gang Ns = Ns Sy 0 Ns | Ns 1 Pq int 3263ff01b23SMartin Matuska.Bl -item -compact 3273ff01b23SMartin Matuska.It 3283ff01b23SMartin MatuskaIf unset, we will first try normal allocation. 3293ff01b23SMartin Matuska.It 3303ff01b23SMartin MatuskaIf that fails then we will do a gang allocation. 3313ff01b23SMartin Matuska.It 3323ff01b23SMartin MatuskaIf that fails then we will do a "try hard" gang allocation. 3333ff01b23SMartin Matuska.It 3343ff01b23SMartin MatuskaIf that fails then we will have a multi-layer gang block. 3353ff01b23SMartin Matuska.El 3363ff01b23SMartin Matuska.Pp 3373ff01b23SMartin Matuska.Bl -item -compact 3383ff01b23SMartin Matuska.It 3393ff01b23SMartin MatuskaIf set, we will first try normal allocation. 3403ff01b23SMartin Matuska.It 3413ff01b23SMartin MatuskaIf that fails then we will do a "try hard" allocation. 3423ff01b23SMartin Matuska.It 3433ff01b23SMartin MatuskaIf that fails we will do a gang allocation. 3443ff01b23SMartin Matuska.It 3453ff01b23SMartin MatuskaIf that fails we will do a "try hard" gang allocation. 3463ff01b23SMartin Matuska.It 3473ff01b23SMartin MatuskaIf that fails then we will have a multi-layer gang block. 3483ff01b23SMartin Matuska.El 3493ff01b23SMartin Matuska. 350be181ee2SMartin Matuska.It Sy zfs_metaslab_find_max_tries Ns = Ns Sy 100 Pq uint 3513ff01b23SMartin MatuskaWhen not trying hard, we only consider this number of the best metaslabs. 3523ff01b23SMartin MatuskaThis improves performance, especially when there are many metaslabs per vdev 3533ff01b23SMartin Matuskaand the allocation can't actually be satisfied 3543ff01b23SMartin Matuska(so we would otherwise iterate all metaslabs). 3553ff01b23SMartin Matuska. 356be181ee2SMartin Matuska.It Sy zfs_vdev_default_ms_count Ns = Ns Sy 200 Pq uint 3573ff01b23SMartin MatuskaWhen a vdev is added, target this number of metaslabs per top-level vdev. 3583ff01b23SMartin Matuska. 359be181ee2SMartin Matuska.It Sy zfs_vdev_default_ms_shift Ns = Ns Sy 29 Po 512 MiB Pc Pq uint 3603ff01b23SMartin MatuskaDefault limit for metaslab size. 3613ff01b23SMartin Matuska. 362dbd5678dSMartin Matuska.It Sy zfs_vdev_max_auto_ashift Ns = Ns Sy 14 Pq uint 363bb2d13b6SMartin MatuskaMaximum ashift used when optimizing for logical \[->] physical sector size on 364bb2d13b6SMartin Matuskanew 3653ff01b23SMartin Matuskatop-level vdevs. 366c7046f76SMartin MatuskaMay be increased up to 367c7046f76SMartin Matuska.Sy ASHIFT_MAX Po 16 Pc , 368c7046f76SMartin Matuskabut this may negatively impact pool space efficiency. 3693ff01b23SMartin Matuska. 370dbd5678dSMartin Matuska.It Sy zfs_vdev_min_auto_ashift Ns = Ns Sy ASHIFT_MIN Po 9 Pc Pq uint 3713ff01b23SMartin MatuskaMinimum ashift used when creating new top-level vdevs. 3723ff01b23SMartin Matuska. 373be181ee2SMartin Matuska.It Sy zfs_vdev_min_ms_count Ns = Ns Sy 16 Pq uint 3743ff01b23SMartin MatuskaMinimum number of metaslabs to create in a top-level vdev. 3753ff01b23SMartin Matuska. 3763ff01b23SMartin Matuska.It Sy vdev_validate_skip Ns = Ns Sy 0 Ns | Ns 1 Pq int 3773ff01b23SMartin MatuskaSkip label validation steps during pool import. 3783ff01b23SMartin MatuskaChanging is not recommended unless you know what you're doing 3793ff01b23SMartin Matuskaand are recovering a damaged label. 3803ff01b23SMartin Matuska. 381be181ee2SMartin Matuska.It Sy zfs_vdev_ms_count_limit Ns = Ns Sy 131072 Po 128k Pc Pq uint 3823ff01b23SMartin MatuskaPractical upper limit of total metaslabs per top-level vdev. 3833ff01b23SMartin Matuska. 3843ff01b23SMartin Matuska.It Sy metaslab_preload_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 3853ff01b23SMartin MatuskaEnable metaslab group preloading. 3863ff01b23SMartin Matuska. 3873ff01b23SMartin Matuska.It Sy metaslab_lba_weighting_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 3883ff01b23SMartin MatuskaGive more weight to metaslabs with lower LBAs, 3893ff01b23SMartin Matuskaassuming they have greater bandwidth, 3903ff01b23SMartin Matuskaas is typically the case on a modern constant angular velocity disk drive. 3913ff01b23SMartin Matuska. 392be181ee2SMartin Matuska.It Sy metaslab_unload_delay Ns = Ns Sy 32 Pq uint 3933ff01b23SMartin MatuskaAfter a metaslab is used, we keep it loaded for this many TXGs, to attempt to 3943ff01b23SMartin Matuskareduce unnecessary reloading. 3953ff01b23SMartin MatuskaNote that both this many TXGs and 3963ff01b23SMartin Matuska.Sy metaslab_unload_delay_ms 3973ff01b23SMartin Matuskamilliseconds must pass before unloading will occur. 3983ff01b23SMartin Matuska. 399be181ee2SMartin Matuska.It Sy metaslab_unload_delay_ms Ns = Ns Sy 600000 Ns ms Po 10 min Pc Pq uint 4003ff01b23SMartin MatuskaAfter a metaslab is used, we keep it loaded for this many milliseconds, 4013ff01b23SMartin Matuskato attempt to reduce unnecessary reloading. 4023ff01b23SMartin MatuskaNote, that both this many milliseconds and 4033ff01b23SMartin Matuska.Sy metaslab_unload_delay 4043ff01b23SMartin MatuskaTXGs must pass before unloading will occur. 4053ff01b23SMartin Matuska. 406be181ee2SMartin Matuska.It Sy reference_history Ns = Ns Sy 3 Pq uint 407bb2d13b6SMartin MatuskaMaximum reference holders being tracked when reference_tracking_enable is 408bb2d13b6SMartin Matuskaactive. 4093ff01b23SMartin Matuska. 4103ff01b23SMartin Matuska.It Sy reference_tracking_enable Ns = Ns Sy 0 Ns | Ns 1 Pq int 4113ff01b23SMartin MatuskaTrack reference holders to 4123ff01b23SMartin Matuska.Sy refcount_t 4133ff01b23SMartin Matuskaobjects (debug builds only). 4143ff01b23SMartin Matuska. 4153ff01b23SMartin Matuska.It Sy send_holes_without_birth_time Ns = Ns Sy 1 Ns | Ns 0 Pq int 4163ff01b23SMartin MatuskaWhen set, the 4173ff01b23SMartin Matuska.Sy hole_birth 4183ff01b23SMartin Matuskaoptimization will not be used, and all holes will always be sent during a 4193ff01b23SMartin Matuska.Nm zfs Cm send . 4203ff01b23SMartin MatuskaThis is useful if you suspect your datasets are affected by a bug in 4213ff01b23SMartin Matuska.Sy hole_birth . 4223ff01b23SMartin Matuska. 4233ff01b23SMartin Matuska.It Sy spa_config_path Ns = Ns Pa /etc/zfs/zpool.cache Pq charp 4243ff01b23SMartin MatuskaSPA config file. 4253ff01b23SMartin Matuska. 426be181ee2SMartin Matuska.It Sy spa_asize_inflation Ns = Ns Sy 24 Pq uint 4273ff01b23SMartin MatuskaMultiplication factor used to estimate actual disk consumption from the 4283ff01b23SMartin Matuskasize of data being written. 4293ff01b23SMartin MatuskaThe default value is a worst case estimate, 4303ff01b23SMartin Matuskabut lower values may be valid for a given pool depending on its configuration. 4313ff01b23SMartin MatuskaPool administrators who understand the factors involved 4323ff01b23SMartin Matuskamay wish to specify a more realistic inflation factor, 4333ff01b23SMartin Matuskaparticularly if they operate close to quota or capacity limits. 4343ff01b23SMartin Matuska. 4353ff01b23SMartin Matuska.It Sy spa_load_print_vdev_tree Ns = Ns Sy 0 Ns | Ns 1 Pq int 436bb2d13b6SMartin MatuskaWhether to print the vdev tree in the debugging message buffer during pool 437bb2d13b6SMartin Matuskaimport. 4383ff01b23SMartin Matuska. 4393ff01b23SMartin Matuska.It Sy spa_load_verify_data Ns = Ns Sy 1 Ns | Ns 0 Pq int 4403ff01b23SMartin MatuskaWhether to traverse data blocks during an "extreme rewind" 4413ff01b23SMartin Matuska.Pq Fl X 4423ff01b23SMartin Matuskaimport. 4433ff01b23SMartin Matuska.Pp 4443ff01b23SMartin MatuskaAn extreme rewind import normally performs a full traversal of all 4453ff01b23SMartin Matuskablocks in the pool for verification. 4463ff01b23SMartin MatuskaIf this parameter is unset, the traversal skips non-metadata blocks. 4473ff01b23SMartin MatuskaIt can be toggled once the 4483ff01b23SMartin Matuskaimport has started to stop or start the traversal of non-metadata blocks. 4493ff01b23SMartin Matuska. 4503ff01b23SMartin Matuska.It Sy spa_load_verify_metadata Ns = Ns Sy 1 Ns | Ns 0 Pq int 4513ff01b23SMartin MatuskaWhether to traverse blocks during an "extreme rewind" 4523ff01b23SMartin Matuska.Pq Fl X 4533ff01b23SMartin Matuskapool import. 4543ff01b23SMartin Matuska.Pp 4553ff01b23SMartin MatuskaAn extreme rewind import normally performs a full traversal of all 4563ff01b23SMartin Matuskablocks in the pool for verification. 4573ff01b23SMartin MatuskaIf this parameter is unset, the traversal is not performed. 4583ff01b23SMartin MatuskaIt can be toggled once the import has started to stop or start the traversal. 4593ff01b23SMartin Matuska. 460be181ee2SMartin Matuska.It Sy spa_load_verify_shift Ns = Ns Sy 4 Po 1/16th Pc Pq uint 4613ff01b23SMartin MatuskaSets the maximum number of bytes to consume during pool import to the log2 4623ff01b23SMartin Matuskafraction of the target ARC size. 4633ff01b23SMartin Matuska. 4643ff01b23SMartin Matuska.It Sy spa_slop_shift Ns = Ns Sy 5 Po 1/32nd Pc Pq int 4653ff01b23SMartin MatuskaNormally, we don't allow the last 4663ff01b23SMartin Matuska.Sy 3.2% Pq Sy 1/2^spa_slop_shift 4673ff01b23SMartin Matuskaof space in the pool to be consumed. 4683ff01b23SMartin MatuskaThis ensures that we don't run the pool completely out of space, 4693ff01b23SMartin Matuskadue to unaccounted changes (e.g. to the MOS). 4703ff01b23SMartin MatuskaIt also limits the worst-case time to allocate space. 4713ff01b23SMartin MatuskaIf we have less than this amount of free space, 4723ff01b23SMartin Matuskamost ZPL operations (e.g. write, create) will return 4733ff01b23SMartin Matuska.Sy ENOSPC . 4743ff01b23SMartin Matuska. 475716fd348SMartin Matuska.It Sy spa_upgrade_errlog_limit Ns = Ns Sy 0 Pq uint 476716fd348SMartin MatuskaLimits the number of on-disk error log entries that will be converted to the 477716fd348SMartin Matuskanew format when enabling the 478716fd348SMartin Matuska.Sy head_errlog 479716fd348SMartin Matuskafeature. 480716fd348SMartin MatuskaThe default is to convert all log entries. 481716fd348SMartin Matuska. 482be181ee2SMartin Matuska.It Sy vdev_removal_max_span Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint 4833ff01b23SMartin MatuskaDuring top-level vdev removal, chunks of data are copied from the vdev 4843ff01b23SMartin Matuskawhich may include free space in order to trade bandwidth for IOPS. 4853ff01b23SMartin MatuskaThis parameter determines the maximum span of free space, in bytes, 4863ff01b23SMartin Matuskawhich will be included as "unnecessary" data in a chunk of copied data. 4873ff01b23SMartin Matuska.Pp 4883ff01b23SMartin MatuskaThe default value here was chosen to align with 4893ff01b23SMartin Matuska.Sy zfs_vdev_read_gap_limit , 4903ff01b23SMartin Matuskawhich is a similar concept when doing 4913ff01b23SMartin Matuskaregular reads (but there's no reason it has to be the same). 4923ff01b23SMartin Matuska. 493dbd5678dSMartin Matuska.It Sy vdev_file_logical_ashift Ns = Ns Sy 9 Po 512 B Pc Pq u64 4943ff01b23SMartin MatuskaLogical ashift for file-based devices. 4953ff01b23SMartin Matuska. 496dbd5678dSMartin Matuska.It Sy vdev_file_physical_ashift Ns = Ns Sy 9 Po 512 B Pc Pq u64 4973ff01b23SMartin MatuskaPhysical ashift for file-based devices. 4983ff01b23SMartin Matuska. 4993ff01b23SMartin Matuska.It Sy zap_iterate_prefetch Ns = Ns Sy 1 Ns | Ns 0 Pq int 5003ff01b23SMartin MatuskaIf set, when we start iterating over a ZAP object, 5013ff01b23SMartin Matuskaprefetch the entire object (all leaf blocks). 5023ff01b23SMartin MatuskaHowever, this is limited by 5033ff01b23SMartin Matuska.Sy dmu_prefetch_max . 5043ff01b23SMartin Matuska. 505*15f0b8c3SMartin Matuska.It Sy zap_micro_max_size Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq int 506*15f0b8c3SMartin MatuskaMaximum micro ZAP size. 507*15f0b8c3SMartin MatuskaA micro ZAP is upgraded to a fat ZAP, once it grows beyond the specified size. 508*15f0b8c3SMartin Matuska. 509dbd5678dSMartin Matuska.It Sy zfetch_array_rd_sz Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 5103ff01b23SMartin MatuskaIf prefetching is enabled, disable prefetching for reads larger than this size. 5113ff01b23SMartin Matuska. 512e3aa18adSMartin Matuska.It Sy zfetch_min_distance Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq uint 513e3aa18adSMartin MatuskaMin bytes to prefetch per stream. 514e3aa18adSMartin MatuskaPrefetch distance starts from the demand access size and quickly grows to 515e3aa18adSMartin Matuskathis value, doubling on each hit. 516e3aa18adSMartin MatuskaAfter that it may grow further by 1/8 per hit, but only if some prefetch 517e3aa18adSMartin Matuskasince last time haven't completed in time to satisfy demand request, i.e. 518e3aa18adSMartin Matuskaprefetch depth didn't cover the read latency or the pool got saturated. 519e3aa18adSMartin Matuska. 520e3aa18adSMartin Matuska.It Sy zfetch_max_distance Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq uint 5213ff01b23SMartin MatuskaMax bytes to prefetch per stream. 5223ff01b23SMartin Matuska. 523716fd348SMartin Matuska.It Sy zfetch_max_idistance Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq uint 5243ff01b23SMartin MatuskaMax bytes to prefetch indirects for per stream. 5253ff01b23SMartin Matuska. 5263ff01b23SMartin Matuska.It Sy zfetch_max_streams Ns = Ns Sy 8 Pq uint 5273ff01b23SMartin MatuskaMax number of streams per zfetch (prefetch streams per file). 5283ff01b23SMartin Matuska. 529e3aa18adSMartin Matuska.It Sy zfetch_min_sec_reap Ns = Ns Sy 1 Pq uint 530e3aa18adSMartin MatuskaMin time before inactive prefetch stream can be reclaimed 531e3aa18adSMartin Matuska. 532e3aa18adSMartin Matuska.It Sy zfetch_max_sec_reap Ns = Ns Sy 2 Pq uint 533e3aa18adSMartin MatuskaMax time before inactive prefetch stream can be deleted 5343ff01b23SMartin Matuska. 5353ff01b23SMartin Matuska.It Sy zfs_abd_scatter_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 5363ff01b23SMartin MatuskaEnables ARC from using scatter/gather lists and forces all allocations to be 5373ff01b23SMartin Matuskalinear in kernel memory. 5383ff01b23SMartin MatuskaDisabling can improve performance in some code paths 5393ff01b23SMartin Matuskaat the expense of fragmented kernel memory. 5403ff01b23SMartin Matuska. 541e92ffd9bSMartin Matuska.It Sy zfs_abd_scatter_max_order Ns = Ns Sy MAX_ORDER\-1 Pq uint 5423ff01b23SMartin MatuskaMaximum number of consecutive memory pages allocated in a single block for 5433ff01b23SMartin Matuskascatter/gather lists. 5443ff01b23SMartin Matuska.Pp 5453ff01b23SMartin MatuskaThe value of 5463ff01b23SMartin Matuska.Sy MAX_ORDER 5473ff01b23SMartin Matuskadepends on kernel configuration. 5483ff01b23SMartin Matuska. 549716fd348SMartin Matuska.It Sy zfs_abd_scatter_min_size Ns = Ns Sy 1536 Ns B Po 1.5 KiB Pc Pq uint 5503ff01b23SMartin MatuskaThis is the minimum allocation size that will use scatter (page-based) ABDs. 5513ff01b23SMartin MatuskaSmaller allocations will use linear ABDs. 5523ff01b23SMartin Matuska. 553dbd5678dSMartin Matuska.It Sy zfs_arc_dnode_limit Ns = Ns Sy 0 Ns B Pq u64 5543ff01b23SMartin MatuskaWhen the number of bytes consumed by dnodes in the ARC exceeds this number of 5553ff01b23SMartin Matuskabytes, try to unpin some of it in response to demand for non-metadata. 5563ff01b23SMartin MatuskaThis value acts as a ceiling to the amount of dnode metadata, and defaults to 5573ff01b23SMartin Matuska.Sy 0 , 5583ff01b23SMartin Matuskawhich indicates that a percent which is based on 5593ff01b23SMartin Matuska.Sy zfs_arc_dnode_limit_percent 5603ff01b23SMartin Matuskaof the ARC meta buffers that may be used for dnodes. 5613ff01b23SMartin Matuska.Pp 5623ff01b23SMartin MatuskaAlso see 5633ff01b23SMartin Matuska.Sy zfs_arc_meta_prune 5643ff01b23SMartin Matuskawhich serves a similar purpose but is used 5653ff01b23SMartin Matuskawhen the amount of metadata in the ARC exceeds 5663ff01b23SMartin Matuska.Sy zfs_arc_meta_limit 5673ff01b23SMartin Matuskarather than in response to overall demand for non-metadata. 5683ff01b23SMartin Matuska. 569dbd5678dSMartin Matuska.It Sy zfs_arc_dnode_limit_percent Ns = Ns Sy 10 Ns % Pq u64 5703ff01b23SMartin MatuskaPercentage that can be consumed by dnodes of ARC meta buffers. 5713ff01b23SMartin Matuska.Pp 5723ff01b23SMartin MatuskaSee also 5733ff01b23SMartin Matuska.Sy zfs_arc_dnode_limit , 5743ff01b23SMartin Matuskawhich serves a similar purpose but has a higher priority if nonzero. 5753ff01b23SMartin Matuska. 576dbd5678dSMartin Matuska.It Sy zfs_arc_dnode_reduce_percent Ns = Ns Sy 10 Ns % Pq u64 5773ff01b23SMartin MatuskaPercentage of ARC dnodes to try to scan in response to demand for non-metadata 5783ff01b23SMartin Matuskawhen the number of bytes consumed by dnodes exceeds 5793ff01b23SMartin Matuska.Sy zfs_arc_dnode_limit . 5803ff01b23SMartin Matuska. 581be181ee2SMartin Matuska.It Sy zfs_arc_average_blocksize Ns = Ns Sy 8192 Ns B Po 8 KiB Pc Pq uint 5823ff01b23SMartin MatuskaThe ARC's buffer hash table is sized based on the assumption of an average 5833ff01b23SMartin Matuskablock size of this value. 584716fd348SMartin MatuskaThis works out to roughly 1 MiB of hash table per 1 GiB of physical memory 5853ff01b23SMartin Matuskawith 8-byte pointers. 5863ff01b23SMartin MatuskaFor configurations with a known larger average block size, 5873ff01b23SMartin Matuskathis value can be increased to reduce the memory footprint. 5883ff01b23SMartin Matuska. 589be181ee2SMartin Matuska.It Sy zfs_arc_eviction_pct Ns = Ns Sy 200 Ns % Pq uint 5903ff01b23SMartin MatuskaWhen 5913ff01b23SMartin Matuska.Fn arc_is_overflowing , 5923ff01b23SMartin Matuska.Fn arc_get_data_impl 5933ff01b23SMartin Matuskawaits for this percent of the requested amount of data to be evicted. 5943ff01b23SMartin MatuskaFor example, by default, for every 595716fd348SMartin Matuska.Em 2 KiB 5963ff01b23SMartin Matuskathat's evicted, 597716fd348SMartin Matuska.Em 1 KiB 5983ff01b23SMartin Matuskaof it may be "reused" by a new allocation. 5993ff01b23SMartin MatuskaSince this is above 6003ff01b23SMartin Matuska.Sy 100 Ns % , 6013ff01b23SMartin Matuskait ensures that progress is made towards getting 6023ff01b23SMartin Matuska.Sy arc_size No under Sy arc_c . 6033ff01b23SMartin MatuskaSince this is finite, it ensures that allocations can still happen, 6043ff01b23SMartin Matuskaeven during the potentially long time that 6053ff01b23SMartin Matuska.Sy arc_size No is more than Sy arc_c . 6063ff01b23SMartin Matuska. 607be181ee2SMartin Matuska.It Sy zfs_arc_evict_batch_limit Ns = Ns Sy 10 Pq uint 6083ff01b23SMartin MatuskaNumber ARC headers to evict per sub-list before proceeding to another sub-list. 6093ff01b23SMartin MatuskaThis batch-style operation prevents entire sub-lists from being evicted at once 6103ff01b23SMartin Matuskabut comes at a cost of additional unlocking and locking. 6113ff01b23SMartin Matuska. 612be181ee2SMartin Matuska.It Sy zfs_arc_grow_retry Ns = Ns Sy 0 Ns s Pq uint 6133ff01b23SMartin MatuskaIf set to a non zero value, it will replace the 6143ff01b23SMartin Matuska.Sy arc_grow_retry 6153ff01b23SMartin Matuskavalue with this value. 6163ff01b23SMartin MatuskaThe 6173ff01b23SMartin Matuska.Sy arc_grow_retry 6183ff01b23SMartin Matuska.No value Pq default Sy 5 Ns s 6193ff01b23SMartin Matuskais the number of seconds the ARC will wait before 6203ff01b23SMartin Matuskatrying to resume growth after a memory pressure event. 6213ff01b23SMartin Matuska. 6223ff01b23SMartin Matuska.It Sy zfs_arc_lotsfree_percent Ns = Ns Sy 10 Ns % Pq int 6233ff01b23SMartin MatuskaThrottle I/O when free system memory drops below this percentage of total 6243ff01b23SMartin Matuskasystem memory. 6253ff01b23SMartin MatuskaSetting this value to 6263ff01b23SMartin Matuska.Sy 0 6273ff01b23SMartin Matuskawill disable the throttle. 6283ff01b23SMartin Matuska. 629dbd5678dSMartin Matuska.It Sy zfs_arc_max Ns = Ns Sy 0 Ns B Pq u64 6303ff01b23SMartin MatuskaMax size of ARC in bytes. 6313ff01b23SMartin MatuskaIf 6323ff01b23SMartin Matuska.Sy 0 , 6333ff01b23SMartin Matuskathen the max size of ARC is determined by the amount of system memory installed. 6343ff01b23SMartin MatuskaUnder Linux, half of system memory will be used as the limit. 6353ff01b23SMartin MatuskaUnder 6363ff01b23SMartin Matuska.Fx , 6373ff01b23SMartin Matuskathe larger of 638716fd348SMartin Matuska.Sy all_system_memory No \- Sy 1 GiB 639e92ffd9bSMartin Matuskaand 640e92ffd9bSMartin Matuska.Sy 5/8 No \(mu Sy all_system_memory 6413ff01b23SMartin Matuskawill be used as the limit. 6423ff01b23SMartin MatuskaThis value must be at least 643716fd348SMartin Matuska.Sy 67108864 Ns B Pq 64 MiB . 6443ff01b23SMartin Matuska.Pp 6453ff01b23SMartin MatuskaThis value can be changed dynamically, with some caveats. 6463ff01b23SMartin MatuskaIt cannot be set back to 6473ff01b23SMartin Matuska.Sy 0 6483ff01b23SMartin Matuskawhile running, and reducing it below the current ARC size will not cause 6493ff01b23SMartin Matuskathe ARC to shrink without memory pressure to induce shrinking. 6503ff01b23SMartin Matuska. 651be181ee2SMartin Matuska.It Sy zfs_arc_meta_adjust_restarts Ns = Ns Sy 4096 Pq uint 6523ff01b23SMartin MatuskaThe number of restart passes to make while scanning the ARC attempting 6533ff01b23SMartin Matuskathe free buffers in order to stay below the 6543ff01b23SMartin Matuska.Sy fs_arc_meta_limit . 6553ff01b23SMartin MatuskaThis value should not need to be tuned but is available to facilitate 6563ff01b23SMartin Matuskaperformance analysis. 6573ff01b23SMartin Matuska. 658dbd5678dSMartin Matuska.It Sy zfs_arc_meta_limit Ns = Ns Sy 0 Ns B Pq u64 6593ff01b23SMartin MatuskaThe maximum allowed size in bytes that metadata buffers are allowed to 6603ff01b23SMartin Matuskaconsume in the ARC. 6613ff01b23SMartin MatuskaWhen this limit is reached, metadata buffers will be reclaimed, 6623ff01b23SMartin Matuskaeven if the overall 6633ff01b23SMartin Matuska.Sy arc_c_max 6643ff01b23SMartin Matuskahas not been reached. 6653ff01b23SMartin MatuskaIt defaults to 6663ff01b23SMartin Matuska.Sy 0 , 6673ff01b23SMartin Matuskawhich indicates that a percentage based on 6683ff01b23SMartin Matuska.Sy zfs_arc_meta_limit_percent 6693ff01b23SMartin Matuskaof the ARC may be used for metadata. 6703ff01b23SMartin Matuska.Pp 671bb2d13b6SMartin MatuskaThis value my be changed dynamically, except that must be set to an explicit 672bb2d13b6SMartin Matuskavalue 6733ff01b23SMartin Matuska.Pq cannot be set back to Sy 0 . 6743ff01b23SMartin Matuska. 675dbd5678dSMartin Matuska.It Sy zfs_arc_meta_limit_percent Ns = Ns Sy 75 Ns % Pq u64 6763ff01b23SMartin MatuskaPercentage of ARC buffers that can be used for metadata. 6773ff01b23SMartin Matuska.Pp 6783ff01b23SMartin MatuskaSee also 6793ff01b23SMartin Matuska.Sy zfs_arc_meta_limit , 6803ff01b23SMartin Matuskawhich serves a similar purpose but has a higher priority if nonzero. 6813ff01b23SMartin Matuska. 682dbd5678dSMartin Matuska.It Sy zfs_arc_meta_min Ns = Ns Sy 0 Ns B Pq u64 6833ff01b23SMartin MatuskaThe minimum allowed size in bytes that metadata buffers may consume in 6843ff01b23SMartin Matuskathe ARC. 6853ff01b23SMartin Matuska. 6863ff01b23SMartin Matuska.It Sy zfs_arc_meta_prune Ns = Ns Sy 10000 Pq int 6873ff01b23SMartin MatuskaThe number of dentries and inodes to be scanned looking for entries 6883ff01b23SMartin Matuskawhich can be dropped. 6893ff01b23SMartin MatuskaThis may be required when the ARC reaches the 6903ff01b23SMartin Matuska.Sy zfs_arc_meta_limit 6913ff01b23SMartin Matuskabecause dentries and inodes can pin buffers in the ARC. 6923ff01b23SMartin MatuskaIncreasing this value will cause to dentry and inode caches 6933ff01b23SMartin Matuskato be pruned more aggressively. 6943ff01b23SMartin MatuskaSetting this value to 6953ff01b23SMartin Matuska.Sy 0 6963ff01b23SMartin Matuskawill disable pruning the inode and dentry caches. 6973ff01b23SMartin Matuska. 698be181ee2SMartin Matuska.It Sy zfs_arc_meta_strategy Ns = Ns Sy 1 Ns | Ns 0 Pq uint 6993ff01b23SMartin MatuskaDefine the strategy for ARC metadata buffer eviction (meta reclaim strategy): 7003ff01b23SMartin Matuska.Bl -tag -compact -offset 4n -width "0 (META_ONLY)" 7013ff01b23SMartin Matuska.It Sy 0 Pq META_ONLY 7023ff01b23SMartin Matuskaevict only the ARC metadata buffers 7033ff01b23SMartin Matuska.It Sy 1 Pq BALANCED 7043ff01b23SMartin Matuskaadditional data buffers may be evicted if required 7053ff01b23SMartin Matuskato evict the required number of metadata buffers. 7063ff01b23SMartin Matuska.El 7073ff01b23SMartin Matuska. 708dbd5678dSMartin Matuska.It Sy zfs_arc_min Ns = Ns Sy 0 Ns B Pq u64 7093ff01b23SMartin MatuskaMin size of ARC in bytes. 7103ff01b23SMartin Matuska.No If set to Sy 0 , arc_c_min 7113ff01b23SMartin Matuskawill default to consuming the larger of 712716fd348SMartin Matuska.Sy 32 MiB 713e92ffd9bSMartin Matuskaand 714e92ffd9bSMartin Matuska.Sy all_system_memory No / Sy 32 . 7153ff01b23SMartin Matuska. 716be181ee2SMartin Matuska.It Sy zfs_arc_min_prefetch_ms Ns = Ns Sy 0 Ns ms Ns Po Ns ≡ Ns 1s Pc Pq uint 7173ff01b23SMartin MatuskaMinimum time prefetched blocks are locked in the ARC. 7183ff01b23SMartin Matuska. 719be181ee2SMartin Matuska.It Sy zfs_arc_min_prescient_prefetch_ms Ns = Ns Sy 0 Ns ms Ns Po Ns ≡ Ns 6s Pc Pq uint 7203ff01b23SMartin MatuskaMinimum time "prescient prefetched" blocks are locked in the ARC. 7213ff01b23SMartin MatuskaThese blocks are meant to be prefetched fairly aggressively ahead of 7223ff01b23SMartin Matuskathe code that may use them. 7233ff01b23SMartin Matuska. 724e92ffd9bSMartin Matuska.It Sy zfs_arc_prune_task_threads Ns = Ns Sy 1 Pq int 725e92ffd9bSMartin MatuskaNumber of arc_prune threads. 726e92ffd9bSMartin Matuska.Fx 727e92ffd9bSMartin Matuskadoes not need more than one. 728e92ffd9bSMartin MatuskaLinux may theoretically use one per mount point up to number of CPUs, 729e92ffd9bSMartin Matuskabut that was not proven to be useful. 730e92ffd9bSMartin Matuska. 7313ff01b23SMartin Matuska.It Sy zfs_max_missing_tvds Ns = Ns Sy 0 Pq int 7323ff01b23SMartin MatuskaNumber of missing top-level vdevs which will be allowed during 7333ff01b23SMartin Matuskapool import (only in read-only mode). 7343ff01b23SMartin Matuska. 735dbd5678dSMartin Matuska.It Sy zfs_max_nvlist_src_size Ns = Sy 0 Pq u64 7363ff01b23SMartin MatuskaMaximum size in bytes allowed to be passed as 7373ff01b23SMartin Matuska.Sy zc_nvlist_src_size 7383ff01b23SMartin Matuskafor ioctls on 7393ff01b23SMartin Matuska.Pa /dev/zfs . 7403ff01b23SMartin MatuskaThis prevents a user from causing the kernel to allocate 7413ff01b23SMartin Matuskaan excessive amount of memory. 7423ff01b23SMartin MatuskaWhen the limit is exceeded, the ioctl fails with 7433ff01b23SMartin Matuska.Sy EINVAL 7443ff01b23SMartin Matuskaand a description of the error is sent to the 7453ff01b23SMartin Matuska.Pa zfs-dbgmsg 7463ff01b23SMartin Matuskalog. 7473ff01b23SMartin MatuskaThis parameter should not need to be touched under normal circumstances. 7483ff01b23SMartin MatuskaIf 7493ff01b23SMartin Matuska.Sy 0 , 7503ff01b23SMartin Matuskaequivalent to a quarter of the user-wired memory limit under 7513ff01b23SMartin Matuska.Fx 7523ff01b23SMartin Matuskaand to 753716fd348SMartin Matuska.Sy 134217728 Ns B Pq 128 MiB 7543ff01b23SMartin Matuskaunder Linux. 7553ff01b23SMartin Matuska. 756be181ee2SMartin Matuska.It Sy zfs_multilist_num_sublists Ns = Ns Sy 0 Pq uint 7573ff01b23SMartin MatuskaTo allow more fine-grained locking, each ARC state contains a series 7583ff01b23SMartin Matuskaof lists for both data and metadata objects. 7593ff01b23SMartin MatuskaLocking is performed at the level of these "sub-lists". 7603ff01b23SMartin MatuskaThis parameters controls the number of sub-lists per ARC state, 7613ff01b23SMartin Matuskaand also applies to other uses of the multilist data structure. 7623ff01b23SMartin Matuska.Pp 7633ff01b23SMartin MatuskaIf 7643ff01b23SMartin Matuska.Sy 0 , 7653ff01b23SMartin Matuskaequivalent to the greater of the number of online CPUs and 7663ff01b23SMartin Matuska.Sy 4 . 7673ff01b23SMartin Matuska. 7683ff01b23SMartin Matuska.It Sy zfs_arc_overflow_shift Ns = Ns Sy 8 Pq int 7693ff01b23SMartin MatuskaThe ARC size is considered to be overflowing if it exceeds the current 7703ff01b23SMartin MatuskaARC target size 7713ff01b23SMartin Matuska.Pq Sy arc_c 7723f9d360cSMartin Matuskaby thresholds determined by this parameter. 7733f9d360cSMartin MatuskaExceeding by 774e92ffd9bSMartin Matuska.Sy ( arc_c No >> Sy zfs_arc_overflow_shift ) No / Sy 2 7753f9d360cSMartin Matuskastarts ARC reclamation process. 7763f9d360cSMartin MatuskaIf that appears insufficient, exceeding by 777e92ffd9bSMartin Matuska.Sy ( arc_c No >> Sy zfs_arc_overflow_shift ) No \(mu Sy 1.5 7783f9d360cSMartin Matuskablocks new buffer allocation until the reclaim thread catches up. 7793f9d360cSMartin MatuskaStarted reclamation process continues till ARC size returns below the 7803f9d360cSMartin Matuskatarget size. 7813ff01b23SMartin Matuska.Pp 7823ff01b23SMartin MatuskaThe default value of 7833ff01b23SMartin Matuska.Sy 8 7843f9d360cSMartin Matuskacauses the ARC to start reclamation if it exceeds the target size by 7853f9d360cSMartin Matuska.Em 0.2% 7863f9d360cSMartin Matuskaof the target size, and block allocations by 7873f9d360cSMartin Matuska.Em 0.6% . 7883ff01b23SMartin Matuska. 789be181ee2SMartin Matuska.It Sy zfs_arc_p_min_shift Ns = Ns Sy 0 Pq uint 7903ff01b23SMartin MatuskaIf nonzero, this will update 7913ff01b23SMartin Matuska.Sy arc_p_min_shift Pq default Sy 4 7923ff01b23SMartin Matuskawith the new value. 7933ff01b23SMartin Matuska.Sy arc_p_min_shift No is used as a shift of Sy arc_c 7943ff01b23SMartin Matuskawhen calculating the minumum 7953ff01b23SMartin Matuska.Sy arc_p No size . 7963ff01b23SMartin Matuska. 7973ff01b23SMartin Matuska.It Sy zfs_arc_p_dampener_disable Ns = Ns Sy 1 Ns | Ns 0 Pq int 7983ff01b23SMartin MatuskaDisable 7993ff01b23SMartin Matuska.Sy arc_p 8003ff01b23SMartin Matuskaadapt dampener, which reduces the maximum single adjustment to 8013ff01b23SMartin Matuska.Sy arc_p . 8023ff01b23SMartin Matuska. 803be181ee2SMartin Matuska.It Sy zfs_arc_shrink_shift Ns = Ns Sy 0 Pq uint 8043ff01b23SMartin MatuskaIf nonzero, this will update 8053ff01b23SMartin Matuska.Sy arc_shrink_shift Pq default Sy 7 8063ff01b23SMartin Matuskawith the new value. 8073ff01b23SMartin Matuska. 8083ff01b23SMartin Matuska.It Sy zfs_arc_pc_percent Ns = Ns Sy 0 Ns % Po off Pc Pq uint 8093ff01b23SMartin MatuskaPercent of pagecache to reclaim ARC to. 8103ff01b23SMartin Matuska.Pp 8113ff01b23SMartin MatuskaThis tunable allows the ZFS ARC to play more nicely 8123ff01b23SMartin Matuskawith the kernel's LRU pagecache. 8133ff01b23SMartin MatuskaIt can guarantee that the ARC size won't collapse under scanning 8143ff01b23SMartin Matuskapressure on the pagecache, yet still allows the ARC to be reclaimed down to 8153ff01b23SMartin Matuska.Sy zfs_arc_min 8163ff01b23SMartin Matuskaif necessary. 8173ff01b23SMartin MatuskaThis value is specified as percent of pagecache size (as measured by 8183ff01b23SMartin Matuska.Sy NR_FILE_PAGES ) , 8193ff01b23SMartin Matuskawhere that percent may exceed 8203ff01b23SMartin Matuska.Sy 100 . 8213ff01b23SMartin MatuskaThis 8223ff01b23SMartin Matuskaonly operates during memory pressure/reclaim. 8233ff01b23SMartin Matuska. 8243ff01b23SMartin Matuska.It Sy zfs_arc_shrinker_limit Ns = Ns Sy 10000 Pq int 8253ff01b23SMartin MatuskaThis is a limit on how many pages the ARC shrinker makes available for 8263ff01b23SMartin Matuskaeviction in response to one page allocation attempt. 8273ff01b23SMartin MatuskaNote that in practice, the kernel's shrinker can ask us to evict 8283ff01b23SMartin Matuskaup to about four times this for one allocation attempt. 8293ff01b23SMartin Matuska.Pp 8303ff01b23SMartin MatuskaThe default limit of 831716fd348SMartin Matuska.Sy 10000 Pq in practice, Em 160 MiB No per allocation attempt with 4 KiB pages 8323ff01b23SMartin Matuskalimits the amount of time spent attempting to reclaim ARC memory to 8333ff01b23SMartin Matuskaless than 100 ms per allocation attempt, 834716fd348SMartin Matuskaeven with a small average compressed block size of ~8 KiB. 8353ff01b23SMartin Matuska.Pp 8363ff01b23SMartin MatuskaThe parameter can be set to 0 (zero) to disable the limit, 8373ff01b23SMartin Matuskaand only applies on Linux. 8383ff01b23SMartin Matuska. 839dbd5678dSMartin Matuska.It Sy zfs_arc_sys_free Ns = Ns Sy 0 Ns B Pq u64 8403ff01b23SMartin MatuskaThe target number of bytes the ARC should leave as free memory on the system. 8413ff01b23SMartin MatuskaIf zero, equivalent to the bigger of 842716fd348SMartin Matuska.Sy 512 KiB No and Sy all_system_memory/64 . 8433ff01b23SMartin Matuska. 8443ff01b23SMartin Matuska.It Sy zfs_autoimport_disable Ns = Ns Sy 1 Ns | Ns 0 Pq int 8453ff01b23SMartin MatuskaDisable pool import at module load by ignoring the cache file 8463ff01b23SMartin Matuska.Pq Sy spa_config_path . 8473ff01b23SMartin Matuska. 8483ff01b23SMartin Matuska.It Sy zfs_checksum_events_per_second Ns = Ns Sy 20 Ns /s Pq uint 8493ff01b23SMartin MatuskaRate limit checksum events to this many per second. 8503ff01b23SMartin MatuskaNote that this should not be set below the ZED thresholds 8513ff01b23SMartin Matuska(currently 10 checksums over 10 seconds) 8523ff01b23SMartin Matuskaor else the daemon may not trigger any action. 8533ff01b23SMartin Matuska. 854be181ee2SMartin Matuska.It Sy zfs_commit_timeout_pct Ns = Ns Sy 5 Ns % Pq uint 8553ff01b23SMartin MatuskaThis controls the amount of time that a ZIL block (lwb) will remain "open" 8563ff01b23SMartin Matuskawhen it isn't "full", and it has a thread waiting for it to be committed to 8573ff01b23SMartin Matuskastable storage. 8583ff01b23SMartin MatuskaThe timeout is scaled based on a percentage of the last lwb 8593ff01b23SMartin Matuskalatency to avoid significantly impacting the latency of each individual 8603ff01b23SMartin Matuskatransaction record (itx). 8613ff01b23SMartin Matuska. 8623ff01b23SMartin Matuska.It Sy zfs_condense_indirect_commit_entry_delay_ms Ns = Ns Sy 0 Ns ms Pq int 8633ff01b23SMartin MatuskaVdev indirection layer (used for device removal) sleeps for this many 8643ff01b23SMartin Matuskamilliseconds during mapping generation. 8653ff01b23SMartin MatuskaIntended for use with the test suite to throttle vdev removal speed. 8663ff01b23SMartin Matuska. 867be181ee2SMartin Matuska.It Sy zfs_condense_indirect_obsolete_pct Ns = Ns Sy 25 Ns % Pq uint 868bb2d13b6SMartin MatuskaMinimum percent of obsolete bytes in vdev mapping required to attempt to 869bb2d13b6SMartin Matuskacondense 8703ff01b23SMartin Matuska.Pq see Sy zfs_condense_indirect_vdevs_enable . 8713ff01b23SMartin MatuskaIntended for use with the test suite 8723ff01b23SMartin Matuskato facilitate triggering condensing as needed. 8733ff01b23SMartin Matuska. 8743ff01b23SMartin Matuska.It Sy zfs_condense_indirect_vdevs_enable Ns = Ns Sy 1 Ns | Ns 0 Pq int 8753ff01b23SMartin MatuskaEnable condensing indirect vdev mappings. 8763ff01b23SMartin MatuskaWhen set, attempt to condense indirect vdev mappings 8773ff01b23SMartin Matuskaif the mapping uses more than 8783ff01b23SMartin Matuska.Sy zfs_condense_min_mapping_bytes 8793ff01b23SMartin Matuskabytes of memory and if the obsolete space map object uses more than 8803ff01b23SMartin Matuska.Sy zfs_condense_max_obsolete_bytes 8813ff01b23SMartin Matuskabytes on-disk. 882bb2d13b6SMartin MatuskaThe condensing process is an attempt to save memory by removing obsolete 883bb2d13b6SMartin Matuskamappings. 8843ff01b23SMartin Matuska. 885dbd5678dSMartin Matuska.It Sy zfs_condense_max_obsolete_bytes Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq u64 8863ff01b23SMartin MatuskaOnly attempt to condense indirect vdev mappings if the on-disk size 8873ff01b23SMartin Matuskaof the obsolete space map object is greater than this number of bytes 8883ff01b23SMartin Matuska.Pq see Sy zfs_condense_indirect_vdevs_enable . 8893ff01b23SMartin Matuska. 890dbd5678dSMartin Matuska.It Sy zfs_condense_min_mapping_bytes Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq u64 8913ff01b23SMartin MatuskaMinimum size vdev mapping to attempt to condense 8923ff01b23SMartin Matuska.Pq see Sy zfs_condense_indirect_vdevs_enable . 8933ff01b23SMartin Matuska. 8943ff01b23SMartin Matuska.It Sy zfs_dbgmsg_enable Ns = Ns Sy 1 Ns | Ns 0 Pq int 8953ff01b23SMartin MatuskaInternally ZFS keeps a small log to facilitate debugging. 8963ff01b23SMartin MatuskaThe log is enabled by default, and can be disabled by unsetting this option. 8973ff01b23SMartin MatuskaThe contents of the log can be accessed by reading 8983ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs/dbgmsg . 8993ff01b23SMartin MatuskaWriting 9003ff01b23SMartin Matuska.Sy 0 9013ff01b23SMartin Matuskato the file clears the log. 9023ff01b23SMartin Matuska.Pp 9033ff01b23SMartin MatuskaThis setting does not influence debug prints due to 9043ff01b23SMartin Matuska.Sy zfs_flags . 9053ff01b23SMartin Matuska. 906be181ee2SMartin Matuska.It Sy zfs_dbgmsg_maxsize Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq uint 9073ff01b23SMartin MatuskaMaximum size of the internal ZFS debug log. 9083ff01b23SMartin Matuska. 9093ff01b23SMartin Matuska.It Sy zfs_dbuf_state_index Ns = Ns Sy 0 Pq int 9103ff01b23SMartin MatuskaHistorically used for controlling what reporting was available under 9113ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs . 9123ff01b23SMartin MatuskaNo effect. 9133ff01b23SMartin Matuska. 9143ff01b23SMartin Matuska.It Sy zfs_deadman_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 9153ff01b23SMartin MatuskaWhen a pool sync operation takes longer than 9163ff01b23SMartin Matuska.Sy zfs_deadman_synctime_ms , 9173ff01b23SMartin Matuskaor when an individual I/O operation takes longer than 9183ff01b23SMartin Matuska.Sy zfs_deadman_ziotime_ms , 9193ff01b23SMartin Matuskathen the operation is considered to be "hung". 9203ff01b23SMartin MatuskaIf 9213ff01b23SMartin Matuska.Sy zfs_deadman_enabled 9223ff01b23SMartin Matuskais set, then the deadman behavior is invoked as described by 9233ff01b23SMartin Matuska.Sy zfs_deadman_failmode . 9243ff01b23SMartin MatuskaBy default, the deadman is enabled and set to 9253ff01b23SMartin Matuska.Sy wait 926c03c5b1cSMartin Matuskawhich results in "hung" I/O operations only being logged. 9273ff01b23SMartin MatuskaThe deadman is automatically disabled when a pool gets suspended. 9283ff01b23SMartin Matuska. 9293ff01b23SMartin Matuska.It Sy zfs_deadman_failmode Ns = Ns Sy wait Pq charp 9303ff01b23SMartin MatuskaControls the failure behavior when the deadman detects a "hung" I/O operation. 9313ff01b23SMartin MatuskaValid values are: 9323ff01b23SMartin Matuska.Bl -tag -compact -offset 4n -width "continue" 9333ff01b23SMartin Matuska.It Sy wait 9343ff01b23SMartin MatuskaWait for a "hung" operation to complete. 9353ff01b23SMartin MatuskaFor each "hung" operation a "deadman" event will be posted 9363ff01b23SMartin Matuskadescribing that operation. 9373ff01b23SMartin Matuska.It Sy continue 9383ff01b23SMartin MatuskaAttempt to recover from a "hung" operation by re-dispatching it 9393ff01b23SMartin Matuskato the I/O pipeline if possible. 9403ff01b23SMartin Matuska.It Sy panic 9413ff01b23SMartin MatuskaPanic the system. 9423ff01b23SMartin MatuskaThis can be used to facilitate automatic fail-over 9433ff01b23SMartin Matuskato a properly configured fail-over partner. 9443ff01b23SMartin Matuska.El 9453ff01b23SMartin Matuska. 946dbd5678dSMartin Matuska.It Sy zfs_deadman_checktime_ms Ns = Ns Sy 60000 Ns ms Po 1 min Pc Pq u64 9473ff01b23SMartin MatuskaCheck time in milliseconds. 9483ff01b23SMartin MatuskaThis defines the frequency at which we check for hung I/O requests 9493ff01b23SMartin Matuskaand potentially invoke the 9503ff01b23SMartin Matuska.Sy zfs_deadman_failmode 9513ff01b23SMartin Matuskabehavior. 9523ff01b23SMartin Matuska. 953dbd5678dSMartin Matuska.It Sy zfs_deadman_synctime_ms Ns = Ns Sy 600000 Ns ms Po 10 min Pc Pq u64 9543ff01b23SMartin MatuskaInterval in milliseconds after which the deadman is triggered and also 9553ff01b23SMartin Matuskathe interval after which a pool sync operation is considered to be "hung". 9563ff01b23SMartin MatuskaOnce this limit is exceeded the deadman will be invoked every 9573ff01b23SMartin Matuska.Sy zfs_deadman_checktime_ms 9583ff01b23SMartin Matuskamilliseconds until the pool sync completes. 9593ff01b23SMartin Matuska. 960dbd5678dSMartin Matuska.It Sy zfs_deadman_ziotime_ms Ns = Ns Sy 300000 Ns ms Po 5 min Pc Pq u64 9613ff01b23SMartin MatuskaInterval in milliseconds after which the deadman is triggered and an 9623ff01b23SMartin Matuskaindividual I/O operation is considered to be "hung". 9633ff01b23SMartin MatuskaAs long as the operation remains "hung", 9643ff01b23SMartin Matuskathe deadman will be invoked every 9653ff01b23SMartin Matuska.Sy zfs_deadman_checktime_ms 9663ff01b23SMartin Matuskamilliseconds until the operation completes. 9673ff01b23SMartin Matuska. 9683ff01b23SMartin Matuska.It Sy zfs_dedup_prefetch Ns = Ns Sy 0 Ns | Ns 1 Pq int 9693ff01b23SMartin MatuskaEnable prefetching dedup-ed blocks which are going to be freed. 9703ff01b23SMartin Matuska. 971be181ee2SMartin Matuska.It Sy zfs_delay_min_dirty_percent Ns = Ns Sy 60 Ns % Pq uint 9723ff01b23SMartin MatuskaStart to delay each transaction once there is this amount of dirty data, 9733ff01b23SMartin Matuskaexpressed as a percentage of 9743ff01b23SMartin Matuska.Sy zfs_dirty_data_max . 9753ff01b23SMartin MatuskaThis value should be at least 9763ff01b23SMartin Matuska.Sy zfs_vdev_async_write_active_max_dirty_percent . 9773ff01b23SMartin Matuska.No See Sx ZFS TRANSACTION DELAY . 9783ff01b23SMartin Matuska. 9793ff01b23SMartin Matuska.It Sy zfs_delay_scale Ns = Ns Sy 500000 Pq int 9803ff01b23SMartin MatuskaThis controls how quickly the transaction delay approaches infinity. 9813ff01b23SMartin MatuskaLarger values cause longer delays for a given amount of dirty data. 9823ff01b23SMartin Matuska.Pp 9833ff01b23SMartin MatuskaFor the smoothest delay, this value should be about 1 billion divided 9843ff01b23SMartin Matuskaby the maximum number of operations per second. 9853ff01b23SMartin MatuskaThis will smoothly handle between ten times and a tenth of this number. 9863ff01b23SMartin Matuska.No See Sx ZFS TRANSACTION DELAY . 9873ff01b23SMartin Matuska.Pp 988e92ffd9bSMartin Matuska.Sy zfs_delay_scale No \(mu Sy zfs_dirty_data_max Em must No be smaller than Sy 2^64 . 9893ff01b23SMartin Matuska. 9903ff01b23SMartin Matuska.It Sy zfs_disable_ivset_guid_check Ns = Ns Sy 0 Ns | Ns 1 Pq int 9913ff01b23SMartin MatuskaDisables requirement for IVset GUIDs to be present and match when doing a raw 9923ff01b23SMartin Matuskareceive of encrypted datasets. 9933ff01b23SMartin MatuskaIntended for users whose pools were created with 9943ff01b23SMartin MatuskaOpenZFS pre-release versions and now have compatibility issues. 9953ff01b23SMartin Matuska. 9963ff01b23SMartin Matuska.It Sy zfs_key_max_salt_uses Ns = Ns Sy 400000000 Po 4*10^8 Pc Pq ulong 9973ff01b23SMartin MatuskaMaximum number of uses of a single salt value before generating a new one for 9983ff01b23SMartin Matuskaencrypted datasets. 9993ff01b23SMartin MatuskaThe default value is also the maximum. 10003ff01b23SMartin Matuska. 10013ff01b23SMartin Matuska.It Sy zfs_object_mutex_size Ns = Ns Sy 64 Pq uint 10023ff01b23SMartin MatuskaSize of the znode hashtable used for holds. 10033ff01b23SMartin Matuska.Pp 10043ff01b23SMartin MatuskaDue to the need to hold locks on objects that may not exist yet, kernel mutexes 10053ff01b23SMartin Matuskaare not created per-object and instead a hashtable is used where collisions 10063ff01b23SMartin Matuskawill result in objects waiting when there is not actually contention on the 10073ff01b23SMartin Matuskasame object. 10083ff01b23SMartin Matuska. 10093ff01b23SMartin Matuska.It Sy zfs_slow_io_events_per_second Ns = Ns Sy 20 Ns /s Pq int 1010bb2d13b6SMartin MatuskaRate limit delay and deadman zevents (which report slow I/O operations) to this 1011bb2d13b6SMartin Matuskamany per 10123ff01b23SMartin Matuskasecond. 10133ff01b23SMartin Matuska. 1014dbd5678dSMartin Matuska.It Sy zfs_unflushed_max_mem_amt Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq u64 10153ff01b23SMartin MatuskaUpper-bound limit for unflushed metadata changes to be held by the 10163ff01b23SMartin Matuskalog spacemap in memory, in bytes. 10173ff01b23SMartin Matuska. 1018dbd5678dSMartin Matuska.It Sy zfs_unflushed_max_mem_ppm Ns = Ns Sy 1000 Ns ppm Po 0.1% Pc Pq u64 10193ff01b23SMartin MatuskaPart of overall system memory that ZFS allows to be used 10203ff01b23SMartin Matuskafor unflushed metadata changes by the log spacemap, in millionths. 10213ff01b23SMartin Matuska. 1022dbd5678dSMartin Matuska.It Sy zfs_unflushed_log_block_max Ns = Ns Sy 131072 Po 128k Pc Pq u64 10233ff01b23SMartin MatuskaDescribes the maximum number of log spacemap blocks allowed for each pool. 10243ff01b23SMartin MatuskaThe default value means that the space in all the log spacemaps 10253ff01b23SMartin Matuskacan add up to no more than 1026716fd348SMartin Matuska.Sy 131072 10273ff01b23SMartin Matuskablocks (which means 1028716fd348SMartin Matuska.Em 16 GiB 10293ff01b23SMartin Matuskaof logical space before compression and ditto blocks, 10303ff01b23SMartin Matuskaassuming that blocksize is 1031716fd348SMartin Matuska.Em 128 KiB ) . 10323ff01b23SMartin Matuska.Pp 10333ff01b23SMartin MatuskaThis tunable is important because it involves a trade-off between import 10343ff01b23SMartin Matuskatime after an unclean export and the frequency of flushing metaslabs. 10353ff01b23SMartin MatuskaThe higher this number is, the more log blocks we allow when the pool is 10363ff01b23SMartin Matuskaactive which means that we flush metaslabs less often and thus decrease 1037c03c5b1cSMartin Matuskathe number of I/O operations for spacemap updates per TXG. 10383ff01b23SMartin MatuskaAt the same time though, that means that in the event of an unclean export, 10393ff01b23SMartin Matuskathere will be more log spacemap blocks for us to read, inducing overhead 10403ff01b23SMartin Matuskain the import time of the pool. 10413ff01b23SMartin MatuskaThe lower the number, the amount of flushing increases, destroying log 10423ff01b23SMartin Matuskablocks quicker as they become obsolete faster, which leaves less blocks 10433ff01b23SMartin Matuskato be read during import time after a crash. 10443ff01b23SMartin Matuska.Pp 10453ff01b23SMartin MatuskaEach log spacemap block existing during pool import leads to approximately 10463ff01b23SMartin Matuskaone extra logical I/O issued. 10473ff01b23SMartin MatuskaThis is the reason why this tunable is exposed in terms of blocks rather 10483ff01b23SMartin Matuskathan space used. 10493ff01b23SMartin Matuska. 1050dbd5678dSMartin Matuska.It Sy zfs_unflushed_log_block_min Ns = Ns Sy 1000 Pq u64 10513ff01b23SMartin MatuskaIf the number of metaslabs is small and our incoming rate is high, 10523ff01b23SMartin Matuskawe could get into a situation that we are flushing all our metaslabs every TXG. 10533ff01b23SMartin MatuskaThus we always allow at least this many log blocks. 10543ff01b23SMartin Matuska. 1055dbd5678dSMartin Matuska.It Sy zfs_unflushed_log_block_pct Ns = Ns Sy 400 Ns % Pq u64 10563ff01b23SMartin MatuskaTunable used to determine the number of blocks that can be used for 10573ff01b23SMartin Matuskathe spacemap log, expressed as a percentage of the total number of 1058716fd348SMartin Matuskaunflushed metaslabs in the pool. 1059716fd348SMartin Matuska. 1060dbd5678dSMartin Matuska.It Sy zfs_unflushed_log_txg_max Ns = Ns Sy 1000 Pq u64 1061716fd348SMartin MatuskaTunable limiting maximum time in TXGs any metaslab may remain unflushed. 1062716fd348SMartin MatuskaIt effectively limits maximum number of unflushed per-TXG spacemap logs 1063716fd348SMartin Matuskathat need to be read after unclean pool export. 10643ff01b23SMartin Matuska. 10653ff01b23SMartin Matuska.It Sy zfs_unlink_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq uint 10663ff01b23SMartin MatuskaWhen enabled, files will not be asynchronously removed from the list of pending 10673ff01b23SMartin Matuskaunlinks and the space they consume will be leaked. 10683ff01b23SMartin MatuskaOnce this option has been disabled and the dataset is remounted, 10693ff01b23SMartin Matuskathe pending unlinks will be processed and the freed space returned to the pool. 10703ff01b23SMartin MatuskaThis option is used by the test suite. 10713ff01b23SMartin Matuska. 10723ff01b23SMartin Matuska.It Sy zfs_delete_blocks Ns = Ns Sy 20480 Pq ulong 10733ff01b23SMartin MatuskaThis is the used to define a large file for the purposes of deletion. 10743ff01b23SMartin MatuskaFiles containing more than 10753ff01b23SMartin Matuska.Sy zfs_delete_blocks 10763ff01b23SMartin Matuskawill be deleted asynchronously, while smaller files are deleted synchronously. 10773ff01b23SMartin MatuskaDecreasing this value will reduce the time spent in an 10783ff01b23SMartin Matuska.Xr unlink 2 1079bb2d13b6SMartin Matuskasystem call, at the expense of a longer delay before the freed space is 1080bb2d13b6SMartin Matuskaavailable. 1081dbd5678dSMartin MatuskaThis only applies on Linux. 10823ff01b23SMartin Matuska. 10833ff01b23SMartin Matuska.It Sy zfs_dirty_data_max Ns = Pq int 10843ff01b23SMartin MatuskaDetermines the dirty space limit in bytes. 10853ff01b23SMartin MatuskaOnce this limit is exceeded, new writes are halted until space frees up. 10863ff01b23SMartin MatuskaThis parameter takes precedence over 10873ff01b23SMartin Matuska.Sy zfs_dirty_data_max_percent . 10883ff01b23SMartin Matuska.No See Sx ZFS TRANSACTION DELAY . 10893ff01b23SMartin Matuska.Pp 10903ff01b23SMartin MatuskaDefaults to 10913ff01b23SMartin Matuska.Sy physical_ram/10 , 10923ff01b23SMartin Matuskacapped at 10933ff01b23SMartin Matuska.Sy zfs_dirty_data_max_max . 10943ff01b23SMartin Matuska. 10953ff01b23SMartin Matuska.It Sy zfs_dirty_data_max_max Ns = Pq int 10963ff01b23SMartin MatuskaMaximum allowable value of 10973ff01b23SMartin Matuska.Sy zfs_dirty_data_max , 10983ff01b23SMartin Matuskaexpressed in bytes. 10993ff01b23SMartin MatuskaThis limit is only enforced at module load time, and will be ignored if 11003ff01b23SMartin Matuska.Sy zfs_dirty_data_max 11013ff01b23SMartin Matuskais later changed. 11023ff01b23SMartin MatuskaThis parameter takes precedence over 11033ff01b23SMartin Matuska.Sy zfs_dirty_data_max_max_percent . 11043ff01b23SMartin Matuska.No See Sx ZFS TRANSACTION DELAY . 11053ff01b23SMartin Matuska.Pp 11063ff01b23SMartin MatuskaDefaults to 1107*15f0b8c3SMartin Matuska.Sy min(physical_ram/4, 4GiB) , 1108*15f0b8c3SMartin Matuskaor 1109*15f0b8c3SMartin Matuska.Sy min(physical_ram/4, 1GiB) 1110*15f0b8c3SMartin Matuskafor 32-bit systems. 11113ff01b23SMartin Matuska. 1112be181ee2SMartin Matuska.It Sy zfs_dirty_data_max_max_percent Ns = Ns Sy 25 Ns % Pq uint 11133ff01b23SMartin MatuskaMaximum allowable value of 11143ff01b23SMartin Matuska.Sy zfs_dirty_data_max , 11153ff01b23SMartin Matuskaexpressed as a percentage of physical RAM. 11163ff01b23SMartin MatuskaThis limit is only enforced at module load time, and will be ignored if 11173ff01b23SMartin Matuska.Sy zfs_dirty_data_max 11183ff01b23SMartin Matuskais later changed. 11193ff01b23SMartin MatuskaThe parameter 11203ff01b23SMartin Matuska.Sy zfs_dirty_data_max_max 11213ff01b23SMartin Matuskatakes precedence over this one. 11223ff01b23SMartin Matuska.No See Sx ZFS TRANSACTION DELAY . 11233ff01b23SMartin Matuska. 1124be181ee2SMartin Matuska.It Sy zfs_dirty_data_max_percent Ns = Ns Sy 10 Ns % Pq uint 11253ff01b23SMartin MatuskaDetermines the dirty space limit, expressed as a percentage of all memory. 11263ff01b23SMartin MatuskaOnce this limit is exceeded, new writes are halted until space frees up. 11273ff01b23SMartin MatuskaThe parameter 11283ff01b23SMartin Matuska.Sy zfs_dirty_data_max 11293ff01b23SMartin Matuskatakes precedence over this one. 11303ff01b23SMartin Matuska.No See Sx ZFS TRANSACTION DELAY . 11313ff01b23SMartin Matuska.Pp 11323ff01b23SMartin MatuskaSubject to 11333ff01b23SMartin Matuska.Sy zfs_dirty_data_max_max . 11343ff01b23SMartin Matuska. 1135be181ee2SMartin Matuska.It Sy zfs_dirty_data_sync_percent Ns = Ns Sy 20 Ns % Pq uint 11363ff01b23SMartin MatuskaStart syncing out a transaction group if there's at least this much dirty data 11373ff01b23SMartin Matuska.Pq as a percentage of Sy zfs_dirty_data_max . 11383ff01b23SMartin MatuskaThis should be less than 11393ff01b23SMartin Matuska.Sy zfs_vdev_async_write_active_min_dirty_percent . 11403ff01b23SMartin Matuska. 11413f9d360cSMartin Matuska.It Sy zfs_wrlog_data_max Ns = Pq int 11423f9d360cSMartin MatuskaThe upper limit of write-transaction zil log data size in bytes. 1143e3aa18adSMartin MatuskaWrite operations are throttled when approaching the limit until log data is 1144e3aa18adSMartin Matuskacleared out after transaction group sync. 1145e3aa18adSMartin MatuskaBecause of some overhead, it should be set at least 2 times the size of 11463f9d360cSMartin Matuska.Sy zfs_dirty_data_max 11473f9d360cSMartin Matuska.No to prevent harming normal write throughput . 11483f9d360cSMartin MatuskaIt also should be smaller than the size of the slog device if slog is present. 11493f9d360cSMartin Matuska.Pp 11503f9d360cSMartin MatuskaDefaults to 11513f9d360cSMartin Matuska.Sy zfs_dirty_data_max*2 11523f9d360cSMartin Matuska. 11533ff01b23SMartin Matuska.It Sy zfs_fallocate_reserve_percent Ns = Ns Sy 110 Ns % Pq uint 11543ff01b23SMartin MatuskaSince ZFS is a copy-on-write filesystem with snapshots, blocks cannot be 11553ff01b23SMartin Matuskapreallocated for a file in order to guarantee that later writes will not 11563ff01b23SMartin Matuskarun out of space. 11573ff01b23SMartin MatuskaInstead, 11583ff01b23SMartin Matuska.Xr fallocate 2 11593ff01b23SMartin Matuskaspace preallocation only checks that sufficient space is currently available 11603ff01b23SMartin Matuskain the pool or the user's project quota allocation, 11613ff01b23SMartin Matuskaand then creates a sparse file of the requested size. 11623ff01b23SMartin MatuskaThe requested space is multiplied by 11633ff01b23SMartin Matuska.Sy zfs_fallocate_reserve_percent 11643ff01b23SMartin Matuskato allow additional space for indirect blocks and other internal metadata. 11653ff01b23SMartin MatuskaSetting this to 11663ff01b23SMartin Matuska.Sy 0 11673ff01b23SMartin Matuskadisables support for 11683ff01b23SMartin Matuska.Xr fallocate 2 11693ff01b23SMartin Matuskaand causes it to return 11703ff01b23SMartin Matuska.Sy EOPNOTSUPP . 11713ff01b23SMartin Matuska. 11723ff01b23SMartin Matuska.It Sy zfs_fletcher_4_impl Ns = Ns Sy fastest Pq string 11733ff01b23SMartin MatuskaSelect a fletcher 4 implementation. 11743ff01b23SMartin Matuska.Pp 11753ff01b23SMartin MatuskaSupported selectors are: 11763ff01b23SMartin Matuska.Sy fastest , scalar , sse2 , ssse3 , avx2 , avx512f , avx512bw , 11773ff01b23SMartin Matuska.No and Sy aarch64_neon . 11783ff01b23SMartin MatuskaAll except 11793ff01b23SMartin Matuska.Sy fastest No and Sy scalar 11803ff01b23SMartin Matuskarequire instruction set extensions to be available, 11813ff01b23SMartin Matuskaand will only appear if ZFS detects that they are present at runtime. 11823ff01b23SMartin MatuskaIf multiple implementations of fletcher 4 are available, the 11833ff01b23SMartin Matuska.Sy fastest 11843ff01b23SMartin Matuskawill be chosen using a micro benchmark. 11853ff01b23SMartin MatuskaSelecting 11863ff01b23SMartin Matuska.Sy scalar 11873ff01b23SMartin Matuskaresults in the original CPU-based calculation being used. 11883ff01b23SMartin MatuskaSelecting any option other than 11893ff01b23SMartin Matuska.Sy fastest No or Sy scalar 11903ff01b23SMartin Matuskaresults in vector instructions 11913ff01b23SMartin Matuskafrom the respective CPU instruction set being used. 11923ff01b23SMartin Matuska. 1193c7046f76SMartin Matuska.It Sy zfs_blake3_impl Ns = Ns Sy fastest Pq string 1194c7046f76SMartin MatuskaSelect a BLAKE3 implementation. 1195c7046f76SMartin Matuska.Pp 1196c7046f76SMartin MatuskaSupported selectors are: 1197c7046f76SMartin Matuska.Sy cycle , fastest , generic , sse2 , sse41 , avx2 , avx512 . 1198c7046f76SMartin MatuskaAll except 1199c7046f76SMartin Matuska.Sy cycle , fastest No and Sy generic 1200c7046f76SMartin Matuskarequire instruction set extensions to be available, 1201c7046f76SMartin Matuskaand will only appear if ZFS detects that they are present at runtime. 1202c7046f76SMartin MatuskaIf multiple implementations of BLAKE3 are available, the 1203c7046f76SMartin Matuska.Sy fastest will be chosen using a micro benchmark. You can see the 1204c7046f76SMartin Matuskabenchmark results by reading this kstat file: 1205c7046f76SMartin Matuska.Pa /proc/spl/kstat/zfs/chksum_bench . 1206c7046f76SMartin Matuska. 12073ff01b23SMartin Matuska.It Sy zfs_free_bpobj_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 12083ff01b23SMartin MatuskaEnable/disable the processing of the free_bpobj object. 12093ff01b23SMartin Matuska. 1210dbd5678dSMartin Matuska.It Sy zfs_async_block_max_blocks Ns = Ns Sy UINT64_MAX Po unlimited Pc Pq u64 12113ff01b23SMartin MatuskaMaximum number of blocks freed in a single TXG. 12123ff01b23SMartin Matuska. 1213dbd5678dSMartin Matuska.It Sy zfs_max_async_dedup_frees Ns = Ns Sy 100000 Po 10^5 Pc Pq u64 12143ff01b23SMartin MatuskaMaximum number of dedup blocks freed in a single TXG. 12153ff01b23SMartin Matuska. 1216be181ee2SMartin Matuska.It Sy zfs_vdev_async_read_max_active Ns = Ns Sy 3 Pq uint 12173ff01b23SMartin MatuskaMaximum asynchronous read I/O operations active to each device. 12183ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12193ff01b23SMartin Matuska. 1220be181ee2SMartin Matuska.It Sy zfs_vdev_async_read_min_active Ns = Ns Sy 1 Pq uint 12213ff01b23SMartin MatuskaMinimum asynchronous read I/O operation active to each device. 12223ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12233ff01b23SMartin Matuska. 1224be181ee2SMartin Matuska.It Sy zfs_vdev_async_write_active_max_dirty_percent Ns = Ns Sy 60 Ns % Pq uint 12253ff01b23SMartin MatuskaWhen the pool has more than this much dirty data, use 12263ff01b23SMartin Matuska.Sy zfs_vdev_async_write_max_active 12273ff01b23SMartin Matuskato limit active async writes. 12283ff01b23SMartin MatuskaIf the dirty data is between the minimum and maximum, 12293ff01b23SMartin Matuskathe active I/O limit is linearly interpolated. 12303ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12313ff01b23SMartin Matuska. 1232be181ee2SMartin Matuska.It Sy zfs_vdev_async_write_active_min_dirty_percent Ns = Ns Sy 30 Ns % Pq uint 12333ff01b23SMartin MatuskaWhen the pool has less than this much dirty data, use 12343ff01b23SMartin Matuska.Sy zfs_vdev_async_write_min_active 12353ff01b23SMartin Matuskato limit active async writes. 12363ff01b23SMartin MatuskaIf the dirty data is between the minimum and maximum, 12373ff01b23SMartin Matuskathe active I/O limit is linearly 12383ff01b23SMartin Matuskainterpolated. 12393ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12403ff01b23SMartin Matuska. 1241bb2d13b6SMartin Matuska.It Sy zfs_vdev_async_write_max_active Ns = Ns Sy 10 Pq uint 12423ff01b23SMartin MatuskaMaximum asynchronous write I/O operations active to each device. 12433ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12443ff01b23SMartin Matuska. 1245be181ee2SMartin Matuska.It Sy zfs_vdev_async_write_min_active Ns = Ns Sy 2 Pq uint 12463ff01b23SMartin MatuskaMinimum asynchronous write I/O operations active to each device. 12473ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12483ff01b23SMartin Matuska.Pp 12493ff01b23SMartin MatuskaLower values are associated with better latency on rotational media but poorer 12503ff01b23SMartin Matuskaresilver performance. 12513ff01b23SMartin MatuskaThe default value of 12523ff01b23SMartin Matuska.Sy 2 12533ff01b23SMartin Matuskawas chosen as a compromise. 12543ff01b23SMartin MatuskaA value of 12553ff01b23SMartin Matuska.Sy 3 12563ff01b23SMartin Matuskahas been shown to improve resilver performance further at a cost of 12573ff01b23SMartin Matuskafurther increasing latency. 12583ff01b23SMartin Matuska. 1259be181ee2SMartin Matuska.It Sy zfs_vdev_initializing_max_active Ns = Ns Sy 1 Pq uint 12603ff01b23SMartin MatuskaMaximum initializing I/O operations active to each device. 12613ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12623ff01b23SMartin Matuska. 1263be181ee2SMartin Matuska.It Sy zfs_vdev_initializing_min_active Ns = Ns Sy 1 Pq uint 12643ff01b23SMartin MatuskaMinimum initializing I/O operations active to each device. 12653ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12663ff01b23SMartin Matuska. 1267be181ee2SMartin Matuska.It Sy zfs_vdev_max_active Ns = Ns Sy 1000 Pq uint 12683ff01b23SMartin MatuskaThe maximum number of I/O operations active to each device. 12693ff01b23SMartin MatuskaIdeally, this will be at least the sum of each queue's 12703ff01b23SMartin Matuska.Sy max_active . 12713ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12723ff01b23SMartin Matuska. 1273dbd5678dSMartin Matuska.It Sy zfs_vdev_open_timeout_ms Ns = Ns Sy 1000 Pq uint 1274dbd5678dSMartin MatuskaTimeout value to wait before determining a device is missing 1275dbd5678dSMartin Matuskaduring import. 1276dbd5678dSMartin MatuskaThis is helpful for transient missing paths due 1277dbd5678dSMartin Matuskato links being briefly removed and recreated in response to 1278dbd5678dSMartin Matuskaudev events. 1279dbd5678dSMartin Matuska. 1280be181ee2SMartin Matuska.It Sy zfs_vdev_rebuild_max_active Ns = Ns Sy 3 Pq uint 12813ff01b23SMartin MatuskaMaximum sequential resilver I/O operations active to each device. 12823ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12833ff01b23SMartin Matuska. 1284be181ee2SMartin Matuska.It Sy zfs_vdev_rebuild_min_active Ns = Ns Sy 1 Pq uint 12853ff01b23SMartin MatuskaMinimum sequential resilver I/O operations active to each device. 12863ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12873ff01b23SMartin Matuska. 1288be181ee2SMartin Matuska.It Sy zfs_vdev_removal_max_active Ns = Ns Sy 2 Pq uint 12893ff01b23SMartin MatuskaMaximum removal I/O operations active to each device. 12903ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12913ff01b23SMartin Matuska. 1292be181ee2SMartin Matuska.It Sy zfs_vdev_removal_min_active Ns = Ns Sy 1 Pq uint 12933ff01b23SMartin MatuskaMinimum removal I/O operations active to each device. 12943ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12953ff01b23SMartin Matuska. 1296be181ee2SMartin Matuska.It Sy zfs_vdev_scrub_max_active Ns = Ns Sy 2 Pq uint 12973ff01b23SMartin MatuskaMaximum scrub I/O operations active to each device. 12983ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 12993ff01b23SMartin Matuska. 1300be181ee2SMartin Matuska.It Sy zfs_vdev_scrub_min_active Ns = Ns Sy 1 Pq uint 13013ff01b23SMartin MatuskaMinimum scrub I/O operations active to each device. 13023ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 13033ff01b23SMartin Matuska. 1304be181ee2SMartin Matuska.It Sy zfs_vdev_sync_read_max_active Ns = Ns Sy 10 Pq uint 13053ff01b23SMartin MatuskaMaximum synchronous read I/O operations active to each device. 13063ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 13073ff01b23SMartin Matuska. 1308be181ee2SMartin Matuska.It Sy zfs_vdev_sync_read_min_active Ns = Ns Sy 10 Pq uint 13093ff01b23SMartin MatuskaMinimum synchronous read I/O operations active to each device. 13103ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 13113ff01b23SMartin Matuska. 1312be181ee2SMartin Matuska.It Sy zfs_vdev_sync_write_max_active Ns = Ns Sy 10 Pq uint 13133ff01b23SMartin MatuskaMaximum synchronous write I/O operations active to each device. 13143ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 13153ff01b23SMartin Matuska. 1316be181ee2SMartin Matuska.It Sy zfs_vdev_sync_write_min_active Ns = Ns Sy 10 Pq uint 13173ff01b23SMartin MatuskaMinimum synchronous write I/O operations active to each device. 13183ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 13193ff01b23SMartin Matuska. 1320be181ee2SMartin Matuska.It Sy zfs_vdev_trim_max_active Ns = Ns Sy 2 Pq uint 13213ff01b23SMartin MatuskaMaximum trim/discard I/O operations active to each device. 13223ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 13233ff01b23SMartin Matuska. 1324be181ee2SMartin Matuska.It Sy zfs_vdev_trim_min_active Ns = Ns Sy 1 Pq uint 13253ff01b23SMartin MatuskaMinimum trim/discard I/O operations active to each device. 13263ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 13273ff01b23SMartin Matuska. 1328be181ee2SMartin Matuska.It Sy zfs_vdev_nia_delay Ns = Ns Sy 5 Pq uint 13293ff01b23SMartin MatuskaFor non-interactive I/O (scrub, resilver, removal, initialize and rebuild), 13303ff01b23SMartin Matuskathe number of concurrently-active I/O operations is limited to 13313ff01b23SMartin Matuska.Sy zfs_*_min_active , 13323ff01b23SMartin Matuskaunless the vdev is "idle". 1333e92ffd9bSMartin MatuskaWhen there are no interactive I/O operations active (synchronous or otherwise), 13343ff01b23SMartin Matuskaand 13353ff01b23SMartin Matuska.Sy zfs_vdev_nia_delay 13363ff01b23SMartin Matuskaoperations have completed since the last interactive operation, 13373ff01b23SMartin Matuskathen the vdev is considered to be "idle", 13383ff01b23SMartin Matuskaand the number of concurrently-active non-interactive operations is increased to 13393ff01b23SMartin Matuska.Sy zfs_*_max_active . 13403ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 13413ff01b23SMartin Matuska. 1342be181ee2SMartin Matuska.It Sy zfs_vdev_nia_credit Ns = Ns Sy 5 Pq uint 13433ff01b23SMartin MatuskaSome HDDs tend to prioritize sequential I/O so strongly, that concurrent 13443ff01b23SMartin Matuskarandom I/O latency reaches several seconds. 13453ff01b23SMartin MatuskaOn some HDDs this happens even if sequential I/O operations 13463ff01b23SMartin Matuskaare submitted one at a time, and so setting 13473ff01b23SMartin Matuska.Sy zfs_*_max_active Ns = Sy 1 13483ff01b23SMartin Matuskadoes not help. 13493ff01b23SMartin MatuskaTo prevent non-interactive I/O, like scrub, 13503ff01b23SMartin Matuskafrom monopolizing the device, no more than 13513ff01b23SMartin Matuska.Sy zfs_vdev_nia_credit operations can be sent 13523ff01b23SMartin Matuskawhile there are outstanding incomplete interactive operations. 13533ff01b23SMartin MatuskaThis enforced wait ensures the HDD services the interactive I/O 13543ff01b23SMartin Matuskawithin a reasonable amount of time. 13553ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 13563ff01b23SMartin Matuska. 1357be181ee2SMartin Matuska.It Sy zfs_vdev_queue_depth_pct Ns = Ns Sy 1000 Ns % Pq uint 13583ff01b23SMartin MatuskaMaximum number of queued allocations per top-level vdev expressed as 13593ff01b23SMartin Matuskaa percentage of 13603ff01b23SMartin Matuska.Sy zfs_vdev_async_write_max_active , 13613ff01b23SMartin Matuskawhich allows the system to detect devices that are more capable 13623ff01b23SMartin Matuskaof handling allocations and to allocate more blocks to those devices. 13633ff01b23SMartin MatuskaThis allows for dynamic allocation distribution when devices are imbalanced, 13643ff01b23SMartin Matuskaas fuller devices will tend to be slower than empty devices. 13653ff01b23SMartin Matuska.Pp 13663ff01b23SMartin MatuskaAlso see 13673ff01b23SMartin Matuska.Sy zio_dva_throttle_enabled . 13683ff01b23SMartin Matuska. 1369dbd5678dSMartin Matuska.It Sy zfs_vdev_failfast_mask Ns = Ns Sy 1 Pq uint 1370dbd5678dSMartin MatuskaDefines if the driver should retire on a given error type. 1371dbd5678dSMartin MatuskaThe following options may be bitwise-ored together: 1372dbd5678dSMartin Matuska.TS 1373dbd5678dSMartin Matuskabox; 1374dbd5678dSMartin Matuskalbz r l l . 1375dbd5678dSMartin Matuska Value Name Description 1376dbd5678dSMartin Matuska_ 1377dbd5678dSMartin Matuska 1 Device No driver retries on device errors 1378dbd5678dSMartin Matuska 2 Transport No driver retries on transport errors. 1379dbd5678dSMartin Matuska 4 Driver No driver retries on driver errors. 1380dbd5678dSMartin Matuska.TE 1381dbd5678dSMartin Matuska. 13823ff01b23SMartin Matuska.It Sy zfs_expire_snapshot Ns = Ns Sy 300 Ns s Pq int 13833ff01b23SMartin MatuskaTime before expiring 13843ff01b23SMartin Matuska.Pa .zfs/snapshot . 13853ff01b23SMartin Matuska. 13863ff01b23SMartin Matuska.It Sy zfs_admin_snapshot Ns = Ns Sy 0 Ns | Ns 1 Pq int 13873ff01b23SMartin MatuskaAllow the creation, removal, or renaming of entries in the 13883ff01b23SMartin Matuska.Sy .zfs/snapshot 13893ff01b23SMartin Matuskadirectory to cause the creation, destruction, or renaming of snapshots. 13903ff01b23SMartin MatuskaWhen enabled, this functionality works both locally and over NFS exports 13913ff01b23SMartin Matuskawhich have the 13923ff01b23SMartin Matuska.Em no_root_squash 13933ff01b23SMartin Matuskaoption set. 13943ff01b23SMartin Matuska. 13953ff01b23SMartin Matuska.It Sy zfs_flags Ns = Ns Sy 0 Pq int 13963ff01b23SMartin MatuskaSet additional debugging flags. 13973ff01b23SMartin MatuskaThe following flags may be bitwise-ored together: 13983ff01b23SMartin Matuska.TS 13993ff01b23SMartin Matuskabox; 14003ff01b23SMartin Matuskalbz r l l . 1401dbd5678dSMartin Matuska Value Name Description 14023ff01b23SMartin Matuska_ 14033ff01b23SMartin Matuska 1 ZFS_DEBUG_DPRINTF Enable dprintf entries in the debug log. 14043ff01b23SMartin Matuska* 2 ZFS_DEBUG_DBUF_VERIFY Enable extra dbuf verifications. 14053ff01b23SMartin Matuska* 4 ZFS_DEBUG_DNODE_VERIFY Enable extra dnode verifications. 14063ff01b23SMartin Matuska 8 ZFS_DEBUG_SNAPNAMES Enable snapshot name verification. 1407*15f0b8c3SMartin Matuska* 16 ZFS_DEBUG_MODIFY Check for illegally modified ARC buffers. 14083ff01b23SMartin Matuska 64 ZFS_DEBUG_ZIO_FREE Enable verification of block frees. 14093ff01b23SMartin Matuska 128 ZFS_DEBUG_HISTOGRAM_VERIFY Enable extra spacemap histogram verifications. 14103ff01b23SMartin Matuska 256 ZFS_DEBUG_METASLAB_VERIFY Verify space accounting on disk matches in-memory \fBrange_trees\fP. 14113ff01b23SMartin Matuska 512 ZFS_DEBUG_SET_ERROR Enable \fBSET_ERROR\fP and dprintf entries in the debug log. 14123ff01b23SMartin Matuska 1024 ZFS_DEBUG_INDIRECT_REMAP Verify split blocks created by device removal. 14133ff01b23SMartin Matuska 2048 ZFS_DEBUG_TRIM Verify TRIM ranges are always within the allocatable range tree. 14143ff01b23SMartin Matuska 4096 ZFS_DEBUG_LOG_SPACEMAP Verify that the log summary is consistent with the spacemap log 14153ff01b23SMartin Matuska and enable \fBzfs_dbgmsgs\fP for metaslab loading and flushing. 14163ff01b23SMartin Matuska.TE 14173ff01b23SMartin Matuska.Sy \& * No Requires debug build . 14183ff01b23SMartin Matuska. 1419c7046f76SMartin Matuska.It Sy zfs_btree_verify_intensity Ns = Ns Sy 0 Pq uint 1420c7046f76SMartin MatuskaEnables btree verification. 1421c7046f76SMartin MatuskaThe following settings are culminative: 1422c7046f76SMartin Matuska.TS 1423c7046f76SMartin Matuskabox; 1424c7046f76SMartin Matuskalbz r l l . 1425c7046f76SMartin Matuska Value Description 1426c7046f76SMartin Matuska 1427c7046f76SMartin Matuska 1 Verify height. 1428c7046f76SMartin Matuska 2 Verify pointers from children to parent. 1429c7046f76SMartin Matuska 3 Verify element counts. 1430c7046f76SMartin Matuska 4 Verify element order. (expensive) 1431c7046f76SMartin Matuska* 5 Verify unused memory is poisoned. (expensive) 1432c7046f76SMartin Matuska.TE 1433c7046f76SMartin Matuska.Sy \& * No Requires debug build . 1434c7046f76SMartin Matuska. 14353ff01b23SMartin Matuska.It Sy zfs_free_leak_on_eio Ns = Ns Sy 0 Ns | Ns 1 Pq int 14363ff01b23SMartin MatuskaIf destroy encounters an 14373ff01b23SMartin Matuska.Sy EIO 14383ff01b23SMartin Matuskawhile reading metadata (e.g. indirect blocks), 14393ff01b23SMartin Matuskaspace referenced by the missing metadata can not be freed. 14403ff01b23SMartin MatuskaNormally this causes the background destroy to become "stalled", 14413ff01b23SMartin Matuskaas it is unable to make forward progress. 14423ff01b23SMartin MatuskaWhile in this stalled state, all remaining space to free 14433ff01b23SMartin Matuskafrom the error-encountering filesystem is "temporarily leaked". 14443ff01b23SMartin MatuskaSet this flag to cause it to ignore the 14453ff01b23SMartin Matuska.Sy EIO , 14463ff01b23SMartin Matuskapermanently leak the space from indirect blocks that can not be read, 14473ff01b23SMartin Matuskaand continue to free everything else that it can. 14483ff01b23SMartin Matuska.Pp 14493ff01b23SMartin MatuskaThe default "stalling" behavior is useful if the storage partially 14503ff01b23SMartin Matuskafails (i.e. some but not all I/O operations fail), and then later recovers. 14513ff01b23SMartin MatuskaIn this case, we will be able to continue pool operations while it is 14523ff01b23SMartin Matuskapartially failed, and when it recovers, we can continue to free the 14533ff01b23SMartin Matuskaspace, with no leaks. 14543ff01b23SMartin MatuskaNote, however, that this case is actually fairly rare. 14553ff01b23SMartin Matuska.Pp 14563ff01b23SMartin MatuskaTypically pools either 14573ff01b23SMartin Matuska.Bl -enum -compact -offset 4n -width "1." 14583ff01b23SMartin Matuska.It 14593ff01b23SMartin Matuskafail completely (but perhaps temporarily, 14603ff01b23SMartin Matuskae.g. due to a top-level vdev going offline), or 14613ff01b23SMartin Matuska.It 14623ff01b23SMartin Matuskahave localized, permanent errors (e.g. disk returns the wrong data 14633ff01b23SMartin Matuskadue to bit flip or firmware bug). 14643ff01b23SMartin Matuska.El 14653ff01b23SMartin MatuskaIn the former case, this setting does not matter because the 14663ff01b23SMartin Matuskapool will be suspended and the sync thread will not be able to make 14673ff01b23SMartin Matuskaforward progress regardless. 14683ff01b23SMartin MatuskaIn the latter, because the error is permanent, the best we can do 14693ff01b23SMartin Matuskais leak the minimum amount of space, 14703ff01b23SMartin Matuskawhich is what setting this flag will do. 14713ff01b23SMartin MatuskaIt is therefore reasonable for this flag to normally be set, 14723ff01b23SMartin Matuskabut we chose the more conservative approach of not setting it, 14733ff01b23SMartin Matuskaso that there is no possibility of 14743ff01b23SMartin Matuskaleaking space in the "partial temporary" failure case. 14753ff01b23SMartin Matuska. 1476be181ee2SMartin Matuska.It Sy zfs_free_min_time_ms Ns = Ns Sy 1000 Ns ms Po 1s Pc Pq uint 14773ff01b23SMartin MatuskaDuring a 14783ff01b23SMartin Matuska.Nm zfs Cm destroy 14793ff01b23SMartin Matuskaoperation using the 14803ff01b23SMartin Matuska.Sy async_destroy 14813ff01b23SMartin Matuskafeature, 14823ff01b23SMartin Matuskaa minimum of this much time will be spent working on freeing blocks per TXG. 14833ff01b23SMartin Matuska. 1484be181ee2SMartin Matuska.It Sy zfs_obsolete_min_time_ms Ns = Ns Sy 500 Ns ms Pq uint 14853ff01b23SMartin MatuskaSimilar to 14863ff01b23SMartin Matuska.Sy zfs_free_min_time_ms , 14873ff01b23SMartin Matuskabut for cleanup of old indirection records for removed vdevs. 14883ff01b23SMartin Matuska. 1489dbd5678dSMartin Matuska.It Sy zfs_immediate_write_sz Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq s64 14903ff01b23SMartin MatuskaLargest data block to write to the ZIL. 14913ff01b23SMartin MatuskaLarger blocks will be treated as if the dataset being written to had the 14923ff01b23SMartin Matuska.Sy logbias Ns = Ns Sy throughput 14933ff01b23SMartin Matuskaproperty set. 14943ff01b23SMartin Matuska. 1495dbd5678dSMartin Matuska.It Sy zfs_initialize_value Ns = Ns Sy 16045690984833335022 Po 0xDEADBEEFDEADBEEE Pc Pq u64 14963ff01b23SMartin MatuskaPattern written to vdev free space by 14973ff01b23SMartin Matuska.Xr zpool-initialize 8 . 14983ff01b23SMartin Matuska. 1499dbd5678dSMartin Matuska.It Sy zfs_initialize_chunk_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 15003ff01b23SMartin MatuskaSize of writes used by 15013ff01b23SMartin Matuska.Xr zpool-initialize 8 . 15023ff01b23SMartin MatuskaThis option is used by the test suite. 15033ff01b23SMartin Matuska. 1504dbd5678dSMartin Matuska.It Sy zfs_livelist_max_entries Ns = Ns Sy 500000 Po 5*10^5 Pc Pq u64 15053ff01b23SMartin MatuskaThe threshold size (in block pointers) at which we create a new sub-livelist. 15063ff01b23SMartin MatuskaLarger sublists are more costly from a memory perspective but the fewer 15073ff01b23SMartin Matuskasublists there are, the lower the cost of insertion. 15083ff01b23SMartin Matuska. 15093ff01b23SMartin Matuska.It Sy zfs_livelist_min_percent_shared Ns = Ns Sy 75 Ns % Pq int 15103ff01b23SMartin MatuskaIf the amount of shared space between a snapshot and its clone drops below 15113ff01b23SMartin Matuskathis threshold, the clone turns off the livelist and reverts to the old 15123ff01b23SMartin Matuskadeletion method. 15133ff01b23SMartin MatuskaThis is in place because livelists no long give us a benefit 15143ff01b23SMartin Matuskaonce a clone has been overwritten enough. 15153ff01b23SMartin Matuska. 15163ff01b23SMartin Matuska.It Sy zfs_livelist_condense_new_alloc Ns = Ns Sy 0 Pq int 15173ff01b23SMartin MatuskaIncremented each time an extra ALLOC blkptr is added to a livelist entry while 15183ff01b23SMartin Matuskait is being condensed. 15193ff01b23SMartin MatuskaThis option is used by the test suite to track race conditions. 15203ff01b23SMartin Matuska. 15213ff01b23SMartin Matuska.It Sy zfs_livelist_condense_sync_cancel Ns = Ns Sy 0 Pq int 15223ff01b23SMartin MatuskaIncremented each time livelist condensing is canceled while in 15233ff01b23SMartin Matuska.Fn spa_livelist_condense_sync . 15243ff01b23SMartin MatuskaThis option is used by the test suite to track race conditions. 15253ff01b23SMartin Matuska. 15263ff01b23SMartin Matuska.It Sy zfs_livelist_condense_sync_pause Ns = Ns Sy 0 Ns | Ns 1 Pq int 15273ff01b23SMartin MatuskaWhen set, the livelist condense process pauses indefinitely before 1528e92ffd9bSMartin Matuskaexecuting the synctask \(em 15293ff01b23SMartin Matuska.Fn spa_livelist_condense_sync . 15303ff01b23SMartin MatuskaThis option is used by the test suite to trigger race conditions. 15313ff01b23SMartin Matuska. 15323ff01b23SMartin Matuska.It Sy zfs_livelist_condense_zthr_cancel Ns = Ns Sy 0 Pq int 15333ff01b23SMartin MatuskaIncremented each time livelist condensing is canceled while in 15343ff01b23SMartin Matuska.Fn spa_livelist_condense_cb . 15353ff01b23SMartin MatuskaThis option is used by the test suite to track race conditions. 15363ff01b23SMartin Matuska. 15373ff01b23SMartin Matuska.It Sy zfs_livelist_condense_zthr_pause Ns = Ns Sy 0 Ns | Ns 1 Pq int 15383ff01b23SMartin MatuskaWhen set, the livelist condense process pauses indefinitely before 15393ff01b23SMartin Matuskaexecuting the open context condensing work in 15403ff01b23SMartin Matuska.Fn spa_livelist_condense_cb . 15413ff01b23SMartin MatuskaThis option is used by the test suite to trigger race conditions. 15423ff01b23SMartin Matuska. 1543dbd5678dSMartin Matuska.It Sy zfs_lua_max_instrlimit Ns = Ns Sy 100000000 Po 10^8 Pc Pq u64 15443ff01b23SMartin MatuskaThe maximum execution time limit that can be set for a ZFS channel program, 15453ff01b23SMartin Matuskaspecified as a number of Lua instructions. 15463ff01b23SMartin Matuska. 1547dbd5678dSMartin Matuska.It Sy zfs_lua_max_memlimit Ns = Ns Sy 104857600 Po 100 MiB Pc Pq u64 15483ff01b23SMartin MatuskaThe maximum memory limit that can be set for a ZFS channel program, specified 15493ff01b23SMartin Matuskain bytes. 15503ff01b23SMartin Matuska. 15513ff01b23SMartin Matuska.It Sy zfs_max_dataset_nesting Ns = Ns Sy 50 Pq int 15523ff01b23SMartin MatuskaThe maximum depth of nested datasets. 15533ff01b23SMartin MatuskaThis value can be tuned temporarily to 15543ff01b23SMartin Matuskafix existing datasets that exceed the predefined limit. 15553ff01b23SMartin Matuska. 1556dbd5678dSMartin Matuska.It Sy zfs_max_log_walking Ns = Ns Sy 5 Pq u64 15573ff01b23SMartin MatuskaThe number of past TXGs that the flushing algorithm of the log spacemap 15583ff01b23SMartin Matuskafeature uses to estimate incoming log blocks. 15593ff01b23SMartin Matuska. 1560dbd5678dSMartin Matuska.It Sy zfs_max_logsm_summary_length Ns = Ns Sy 10 Pq u64 15613ff01b23SMartin MatuskaMaximum number of rows allowed in the summary of the spacemap log. 15623ff01b23SMartin Matuska. 1563be181ee2SMartin Matuska.It Sy zfs_max_recordsize Ns = Ns Sy 16777216 Po 16 MiB Pc Pq uint 15643ff01b23SMartin MatuskaWe currently support block sizes from 1565716fd348SMartin Matuska.Em 512 Po 512 B Pc No to Em 16777216 Po 16 MiB Pc . 15663ff01b23SMartin MatuskaThe benefits of larger blocks, and thus larger I/O, 15673ff01b23SMartin Matuskaneed to be weighed against the cost of COWing a giant block to modify one byte. 15683ff01b23SMartin MatuskaAdditionally, very large blocks can have an impact on I/O latency, 15693ff01b23SMartin Matuskaand also potentially on the memory allocator. 1570716fd348SMartin MatuskaTherefore, we formerly forbade creating blocks larger than 1M. 1571716fd348SMartin MatuskaLarger blocks could be created by changing it, 15723ff01b23SMartin Matuskaand pools with larger blocks can always be imported and used, 15733ff01b23SMartin Matuskaregardless of this setting. 15743ff01b23SMartin Matuska. 15753ff01b23SMartin Matuska.It Sy zfs_allow_redacted_dataset_mount Ns = Ns Sy 0 Ns | Ns 1 Pq int 15763ff01b23SMartin MatuskaAllow datasets received with redacted send/receive to be mounted. 15773ff01b23SMartin MatuskaNormally disabled because these datasets may be missing key data. 15783ff01b23SMartin Matuska. 1579dbd5678dSMartin Matuska.It Sy zfs_min_metaslabs_to_flush Ns = Ns Sy 1 Pq u64 15803ff01b23SMartin MatuskaMinimum number of metaslabs to flush per dirty TXG. 15813ff01b23SMartin Matuska. 1582be181ee2SMartin Matuska.It Sy zfs_metaslab_fragmentation_threshold Ns = Ns Sy 70 Ns % Pq uint 15833ff01b23SMartin MatuskaAllow metaslabs to keep their active state as long as their fragmentation 15843ff01b23SMartin Matuskapercentage is no more than this value. 15853ff01b23SMartin MatuskaAn active metaslab that exceeds this threshold 15863ff01b23SMartin Matuskawill no longer keep its active status allowing better metaslabs to be selected. 15873ff01b23SMartin Matuska. 1588be181ee2SMartin Matuska.It Sy zfs_mg_fragmentation_threshold Ns = Ns Sy 95 Ns % Pq uint 15893ff01b23SMartin MatuskaMetaslab groups are considered eligible for allocations if their 15903ff01b23SMartin Matuskafragmentation metric (measured as a percentage) is less than or equal to 15913ff01b23SMartin Matuskathis value. 15923ff01b23SMartin MatuskaIf a metaslab group exceeds this threshold then it will be 15933ff01b23SMartin Matuskaskipped unless all metaslab groups within the metaslab class have also 15943ff01b23SMartin Matuskacrossed this threshold. 15953ff01b23SMartin Matuska. 1596be181ee2SMartin Matuska.It Sy zfs_mg_noalloc_threshold Ns = Ns Sy 0 Ns % Pq uint 15973ff01b23SMartin MatuskaDefines a threshold at which metaslab groups should be eligible for allocations. 15983ff01b23SMartin MatuskaThe value is expressed as a percentage of free space 15993ff01b23SMartin Matuskabeyond which a metaslab group is always eligible for allocations. 16003ff01b23SMartin MatuskaIf a metaslab group's free space is less than or equal to the 16013ff01b23SMartin Matuskathreshold, the allocator will avoid allocating to that group 16023ff01b23SMartin Matuskaunless all groups in the pool have reached the threshold. 16033ff01b23SMartin MatuskaOnce all groups have reached the threshold, all groups are allowed to accept 16043ff01b23SMartin Matuskaallocations. 16053ff01b23SMartin MatuskaThe default value of 16063ff01b23SMartin Matuska.Sy 0 1607bb2d13b6SMartin Matuskadisables the feature and causes all metaslab groups to be eligible for 1608bb2d13b6SMartin Matuskaallocations. 16093ff01b23SMartin Matuska.Pp 16103ff01b23SMartin MatuskaThis parameter allows one to deal with pools having heavily imbalanced 16113ff01b23SMartin Matuskavdevs such as would be the case when a new vdev has been added. 16123ff01b23SMartin MatuskaSetting the threshold to a non-zero percentage will stop allocations 16133ff01b23SMartin Matuskafrom being made to vdevs that aren't filled to the specified percentage 16143ff01b23SMartin Matuskaand allow lesser filled vdevs to acquire more allocations than they 16153ff01b23SMartin Matuskaotherwise would under the old 16163ff01b23SMartin Matuska.Sy zfs_mg_alloc_failures 16173ff01b23SMartin Matuskafacility. 16183ff01b23SMartin Matuska. 16193ff01b23SMartin Matuska.It Sy zfs_ddt_data_is_special Ns = Ns Sy 1 Ns | Ns 0 Pq int 16203ff01b23SMartin MatuskaIf enabled, ZFS will place DDT data into the special allocation class. 16213ff01b23SMartin Matuska. 16223ff01b23SMartin Matuska.It Sy zfs_user_indirect_is_special Ns = Ns Sy 1 Ns | Ns 0 Pq int 16233ff01b23SMartin MatuskaIf enabled, ZFS will place user data indirect blocks 16243ff01b23SMartin Matuskainto the special allocation class. 16253ff01b23SMartin Matuska. 1626be181ee2SMartin Matuska.It Sy zfs_multihost_history Ns = Ns Sy 0 Pq uint 1627bb2d13b6SMartin MatuskaHistorical statistics for this many latest multihost updates will be available 1628bb2d13b6SMartin Matuskain 16293ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /multihost . 16303ff01b23SMartin Matuska. 1631dbd5678dSMartin Matuska.It Sy zfs_multihost_interval Ns = Ns Sy 1000 Ns ms Po 1 s Pc Pq u64 16323ff01b23SMartin MatuskaUsed to control the frequency of multihost writes which are performed when the 16333ff01b23SMartin Matuska.Sy multihost 16343ff01b23SMartin Matuskapool property is on. 16353ff01b23SMartin MatuskaThis is one of the factors used to determine the 16363ff01b23SMartin Matuskalength of the activity check during import. 16373ff01b23SMartin Matuska.Pp 16383ff01b23SMartin MatuskaThe multihost write period is 1639e92ffd9bSMartin Matuska.Sy zfs_multihost_interval No / Sy leaf-vdevs . 16403ff01b23SMartin MatuskaOn average a multihost write will be issued for each leaf vdev 16413ff01b23SMartin Matuskaevery 16423ff01b23SMartin Matuska.Sy zfs_multihost_interval 16433ff01b23SMartin Matuskamilliseconds. 16443ff01b23SMartin MatuskaIn practice, the observed period can vary with the I/O load 16453ff01b23SMartin Matuskaand this observed value is the delay which is stored in the uberblock. 16463ff01b23SMartin Matuska. 16473ff01b23SMartin Matuska.It Sy zfs_multihost_import_intervals Ns = Ns Sy 20 Pq uint 16483ff01b23SMartin MatuskaUsed to control the duration of the activity test on import. 16493ff01b23SMartin MatuskaSmaller values of 16503ff01b23SMartin Matuska.Sy zfs_multihost_import_intervals 16513ff01b23SMartin Matuskawill reduce the import time but increase 16523ff01b23SMartin Matuskathe risk of failing to detect an active pool. 16533ff01b23SMartin MatuskaThe total activity check time is never allowed to drop below one second. 16543ff01b23SMartin Matuska.Pp 16553ff01b23SMartin MatuskaOn import the activity check waits a minimum amount of time determined by 1656e92ffd9bSMartin Matuska.Sy zfs_multihost_interval No \(mu Sy zfs_multihost_import_intervals , 16573ff01b23SMartin Matuskaor the same product computed on the host which last had the pool imported, 16583ff01b23SMartin Matuskawhichever is greater. 16593ff01b23SMartin MatuskaThe activity check time may be further extended if the value of MMP 16603ff01b23SMartin Matuskadelay found in the best uberblock indicates actual multihost updates happened 16613ff01b23SMartin Matuskaat longer intervals than 16623ff01b23SMartin Matuska.Sy zfs_multihost_interval . 16633ff01b23SMartin MatuskaA minimum of 16643ff01b23SMartin Matuska.Em 100 ms 16653ff01b23SMartin Matuskais enforced. 16663ff01b23SMartin Matuska.Pp 16673ff01b23SMartin Matuska.Sy 0 No is equivalent to Sy 1 . 16683ff01b23SMartin Matuska. 16693ff01b23SMartin Matuska.It Sy zfs_multihost_fail_intervals Ns = Ns Sy 10 Pq uint 16703ff01b23SMartin MatuskaControls the behavior of the pool when multihost write failures or delays are 16713ff01b23SMartin Matuskadetected. 16723ff01b23SMartin Matuska.Pp 16733ff01b23SMartin MatuskaWhen 16743ff01b23SMartin Matuska.Sy 0 , 16753ff01b23SMartin Matuskamultihost write failures or delays are ignored. 16763ff01b23SMartin MatuskaThe failures will still be reported to the ZED which depending on 16773ff01b23SMartin Matuskaits configuration may take action such as suspending the pool or offlining a 16783ff01b23SMartin Matuskadevice. 16793ff01b23SMartin Matuska.Pp 16803ff01b23SMartin MatuskaOtherwise, the pool will be suspended if 1681e92ffd9bSMartin Matuska.Sy zfs_multihost_fail_intervals No \(mu Sy zfs_multihost_interval 16823ff01b23SMartin Matuskamilliseconds pass without a successful MMP write. 16833ff01b23SMartin MatuskaThis guarantees the activity test will see MMP writes if the pool is imported. 16843ff01b23SMartin Matuska.Sy 1 No is equivalent to Sy 2 ; 16853ff01b23SMartin Matuskathis is necessary to prevent the pool from being suspended 16863ff01b23SMartin Matuskadue to normal, small I/O latency variations. 16873ff01b23SMartin Matuska. 16883ff01b23SMartin Matuska.It Sy zfs_no_scrub_io Ns = Ns Sy 0 Ns | Ns 1 Pq int 16893ff01b23SMartin MatuskaSet to disable scrub I/O. 16903ff01b23SMartin MatuskaThis results in scrubs not actually scrubbing data and 16913ff01b23SMartin Matuskasimply doing a metadata crawl of the pool instead. 16923ff01b23SMartin Matuska. 16933ff01b23SMartin Matuska.It Sy zfs_no_scrub_prefetch Ns = Ns Sy 0 Ns | Ns 1 Pq int 16943ff01b23SMartin MatuskaSet to disable block prefetching for scrubs. 16953ff01b23SMartin Matuska. 16963ff01b23SMartin Matuska.It Sy zfs_nocacheflush Ns = Ns Sy 0 Ns | Ns 1 Pq int 16973ff01b23SMartin MatuskaDisable cache flush operations on disks when writing. 16983ff01b23SMartin MatuskaSetting this will cause pool corruption on power loss 16993ff01b23SMartin Matuskaif a volatile out-of-order write cache is enabled. 17003ff01b23SMartin Matuska. 17013ff01b23SMartin Matuska.It Sy zfs_nopwrite_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 17023ff01b23SMartin MatuskaAllow no-operation writes. 17033ff01b23SMartin MatuskaThe occurrence of nopwrites will further depend on other pool properties 17043ff01b23SMartin Matuska.Pq i.a. the checksumming and compression algorithms . 17053ff01b23SMartin Matuska. 1706681ce946SMartin Matuska.It Sy zfs_dmu_offset_next_sync Ns = Ns Sy 1 Ns | Ns 0 Pq int 17073ff01b23SMartin MatuskaEnable forcing TXG sync to find holes. 1708681ce946SMartin MatuskaWhen enabled forces ZFS to sync data when 17093ff01b23SMartin Matuska.Sy SEEK_HOLE No or Sy SEEK_DATA 1710681ce946SMartin Matuskaflags are used allowing holes in a file to be accurately reported. 1711681ce946SMartin MatuskaWhen disabled holes will not be reported in recently dirtied files. 17123ff01b23SMartin Matuska. 1713716fd348SMartin Matuska.It Sy zfs_pd_bytes_max Ns = Ns Sy 52428800 Ns B Po 50 MiB Pc Pq int 17143ff01b23SMartin MatuskaThe number of bytes which should be prefetched during a pool traversal, like 17153ff01b23SMartin Matuska.Nm zfs Cm send 17163ff01b23SMartin Matuskaor other data crawling operations. 17173ff01b23SMartin Matuska. 1718be181ee2SMartin Matuska.It Sy zfs_traverse_indirect_prefetch_limit Ns = Ns Sy 32 Pq uint 17193ff01b23SMartin MatuskaThe number of blocks pointed by indirect (non-L0) block which should be 17203ff01b23SMartin Matuskaprefetched during a pool traversal, like 17213ff01b23SMartin Matuska.Nm zfs Cm send 17223ff01b23SMartin Matuskaor other data crawling operations. 17233ff01b23SMartin Matuska. 1724dbd5678dSMartin Matuska.It Sy zfs_per_txg_dirty_frees_percent Ns = Ns Sy 30 Ns % Pq u64 17253ff01b23SMartin MatuskaControl percentage of dirtied indirect blocks from frees allowed into one TXG. 17263ff01b23SMartin MatuskaAfter this threshold is crossed, additional frees will wait until the next TXG. 17273ff01b23SMartin Matuska.Sy 0 No disables this throttle . 17283ff01b23SMartin Matuska. 17293ff01b23SMartin Matuska.It Sy zfs_prefetch_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int 17303ff01b23SMartin MatuskaDisable predictive prefetch. 1731c03c5b1cSMartin MatuskaNote that it leaves "prescient" prefetch 1732c03c5b1cSMartin Matuska.Pq for, e.g., Nm zfs Cm send 17333ff01b23SMartin Matuskaintact. 17343ff01b23SMartin MatuskaUnlike predictive prefetch, prescient prefetch never issues I/O 17353ff01b23SMartin Matuskathat ends up not being needed, so it can't hurt performance. 17363ff01b23SMartin Matuska. 17373ff01b23SMartin Matuska.It Sy zfs_qat_checksum_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int 17383ff01b23SMartin MatuskaDisable QAT hardware acceleration for SHA256 checksums. 17393ff01b23SMartin MatuskaMay be unset after the ZFS modules have been loaded to initialize the QAT 17403ff01b23SMartin Matuskahardware as long as support is compiled in and the QAT driver is present. 17413ff01b23SMartin Matuska. 17423ff01b23SMartin Matuska.It Sy zfs_qat_compress_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int 17433ff01b23SMartin MatuskaDisable QAT hardware acceleration for gzip compression. 17443ff01b23SMartin MatuskaMay be unset after the ZFS modules have been loaded to initialize the QAT 17453ff01b23SMartin Matuskahardware as long as support is compiled in and the QAT driver is present. 17463ff01b23SMartin Matuska. 17473ff01b23SMartin Matuska.It Sy zfs_qat_encrypt_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int 17483ff01b23SMartin MatuskaDisable QAT hardware acceleration for AES-GCM encryption. 17493ff01b23SMartin MatuskaMay be unset after the ZFS modules have been loaded to initialize the QAT 17503ff01b23SMartin Matuskahardware as long as support is compiled in and the QAT driver is present. 17513ff01b23SMartin Matuska. 1752dbd5678dSMartin Matuska.It Sy zfs_vnops_read_chunk_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 17533ff01b23SMartin MatuskaBytes to read per chunk. 17543ff01b23SMartin Matuska. 1755be181ee2SMartin Matuska.It Sy zfs_read_history Ns = Ns Sy 0 Pq uint 17563ff01b23SMartin MatuskaHistorical statistics for this many latest reads will be available in 17573ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /reads . 17583ff01b23SMartin Matuska. 17593ff01b23SMartin Matuska.It Sy zfs_read_history_hits Ns = Ns Sy 0 Ns | Ns 1 Pq int 17603ff01b23SMartin MatuskaInclude cache hits in read history 17613ff01b23SMartin Matuska. 1762dbd5678dSMartin Matuska.It Sy zfs_rebuild_max_segment Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 17633ff01b23SMartin MatuskaMaximum read segment size to issue when sequentially resilvering a 17643ff01b23SMartin Matuskatop-level vdev. 17653ff01b23SMartin Matuska. 17663ff01b23SMartin Matuska.It Sy zfs_rebuild_scrub_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 17673ff01b23SMartin MatuskaAutomatically start a pool scrub when the last active sequential resilver 17683ff01b23SMartin Matuskacompletes in order to verify the checksums of all blocks which have been 17693ff01b23SMartin Matuskaresilvered. 17703ff01b23SMartin MatuskaThis is enabled by default and strongly recommended. 17713ff01b23SMartin Matuska. 1772dbd5678dSMartin Matuska.It Sy zfs_rebuild_vdev_limit Ns = Ns Sy 33554432 Ns B Po 32 MiB Pc Pq u64 17733ff01b23SMartin MatuskaMaximum amount of I/O that can be concurrently issued for a sequential 17743ff01b23SMartin Matuskaresilver per leaf device, given in bytes. 17753ff01b23SMartin Matuska. 17763ff01b23SMartin Matuska.It Sy zfs_reconstruct_indirect_combinations_max Ns = Ns Sy 4096 Pq int 17773ff01b23SMartin MatuskaIf an indirect split block contains more than this many possible unique 17783ff01b23SMartin Matuskacombinations when being reconstructed, consider it too computationally 17793ff01b23SMartin Matuskaexpensive to check them all. 17803ff01b23SMartin MatuskaInstead, try at most this many randomly selected 17813ff01b23SMartin Matuskacombinations each time the block is accessed. 17823ff01b23SMartin MatuskaThis allows all segment copies to participate fairly 17833ff01b23SMartin Matuskain the reconstruction when all combinations 17843ff01b23SMartin Matuskacannot be checked and prevents repeated use of one bad copy. 17853ff01b23SMartin Matuska. 17863ff01b23SMartin Matuska.It Sy zfs_recover Ns = Ns Sy 0 Ns | Ns 1 Pq int 17873ff01b23SMartin MatuskaSet to attempt to recover from fatal errors. 17883ff01b23SMartin MatuskaThis should only be used as a last resort, 17893ff01b23SMartin Matuskaas it typically results in leaked space, or worse. 17903ff01b23SMartin Matuska. 17913ff01b23SMartin Matuska.It Sy zfs_removal_ignore_errors Ns = Ns Sy 0 Ns | Ns 1 Pq int 1792c03c5b1cSMartin MatuskaIgnore hard I/O errors during device removal. 1793c03c5b1cSMartin MatuskaWhen set, if a device encounters a hard I/O error during the removal process 17943ff01b23SMartin Matuskathe removal will not be cancelled. 17953ff01b23SMartin MatuskaThis can result in a normally recoverable block becoming permanently damaged 17963ff01b23SMartin Matuskaand is hence not recommended. 17973ff01b23SMartin MatuskaThis should only be used as a last resort when the 17983ff01b23SMartin Matuskapool cannot be returned to a healthy state prior to removing the device. 17993ff01b23SMartin Matuska. 1800be181ee2SMartin Matuska.It Sy zfs_removal_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq uint 18013ff01b23SMartin MatuskaThis is used by the test suite so that it can ensure that certain actions 18023ff01b23SMartin Matuskahappen while in the middle of a removal. 18033ff01b23SMartin Matuska. 1804be181ee2SMartin Matuska.It Sy zfs_remove_max_segment Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint 18053ff01b23SMartin MatuskaThe largest contiguous segment that we will attempt to allocate when removing 18063ff01b23SMartin Matuskaa device. 18073ff01b23SMartin MatuskaIf there is a performance problem with attempting to allocate large blocks, 18083ff01b23SMartin Matuskaconsider decreasing this. 18093ff01b23SMartin MatuskaThe default value is also the maximum. 18103ff01b23SMartin Matuska. 18113ff01b23SMartin Matuska.It Sy zfs_resilver_disable_defer Ns = Ns Sy 0 Ns | Ns 1 Pq int 18123ff01b23SMartin MatuskaIgnore the 18133ff01b23SMartin Matuska.Sy resilver_defer 18143ff01b23SMartin Matuskafeature, causing an operation that would start a resilver to 18153ff01b23SMartin Matuskaimmediately restart the one in progress. 18163ff01b23SMartin Matuska. 1817be181ee2SMartin Matuska.It Sy zfs_resilver_min_time_ms Ns = Ns Sy 3000 Ns ms Po 3 s Pc Pq uint 18183ff01b23SMartin MatuskaResilvers are processed by the sync thread. 18193ff01b23SMartin MatuskaWhile resilvering, it will spend at least this much time 18203ff01b23SMartin Matuskaworking on a resilver between TXG flushes. 18213ff01b23SMartin Matuska. 18223ff01b23SMartin Matuska.It Sy zfs_scan_ignore_errors Ns = Ns Sy 0 Ns | Ns 1 Pq int 18233ff01b23SMartin MatuskaIf set, remove the DTL (dirty time list) upon completion of a pool scan (scrub), 18243ff01b23SMartin Matuskaeven if there were unrepairable errors. 18253ff01b23SMartin MatuskaIntended to be used during pool repair or recovery to 18263ff01b23SMartin Matuskastop resilvering when the pool is next imported. 18273ff01b23SMartin Matuska. 1828be181ee2SMartin Matuska.It Sy zfs_scrub_min_time_ms Ns = Ns Sy 1000 Ns ms Po 1 s Pc Pq uint 18293ff01b23SMartin MatuskaScrubs are processed by the sync thread. 18303ff01b23SMartin MatuskaWhile scrubbing, it will spend at least this much time 18313ff01b23SMartin Matuskaworking on a scrub between TXG flushes. 18323ff01b23SMartin Matuska. 1833be181ee2SMartin Matuska.It Sy zfs_scan_checkpoint_intval Ns = Ns Sy 7200 Ns s Po 2 hour Pc Pq uint 18343ff01b23SMartin MatuskaTo preserve progress across reboots, the sequential scan algorithm periodically 18353ff01b23SMartin Matuskaneeds to stop metadata scanning and issue all the verification I/O to disk. 18363ff01b23SMartin MatuskaThe frequency of this flushing is determined by this tunable. 18373ff01b23SMartin Matuska. 1838be181ee2SMartin Matuska.It Sy zfs_scan_fill_weight Ns = Ns Sy 3 Pq uint 18393ff01b23SMartin MatuskaThis tunable affects how scrub and resilver I/O segments are ordered. 18403ff01b23SMartin MatuskaA higher number indicates that we care more about how filled in a segment is, 18413ff01b23SMartin Matuskawhile a lower number indicates we care more about the size of the extent without 18423ff01b23SMartin Matuskaconsidering the gaps within a segment. 18433ff01b23SMartin MatuskaThis value is only tunable upon module insertion. 1844bb2d13b6SMartin MatuskaChanging the value afterwards will have no effect on scrub or resilver 1845bb2d13b6SMartin Matuskaperformance. 18463ff01b23SMartin Matuska. 1847be181ee2SMartin Matuska.It Sy zfs_scan_issue_strategy Ns = Ns Sy 0 Pq uint 18483ff01b23SMartin MatuskaDetermines the order that data will be verified while scrubbing or resilvering: 18493ff01b23SMartin Matuska.Bl -tag -compact -offset 4n -width "a" 18503ff01b23SMartin Matuska.It Sy 1 18513ff01b23SMartin MatuskaData will be verified as sequentially as possible, given the 18523ff01b23SMartin Matuskaamount of memory reserved for scrubbing 18533ff01b23SMartin Matuska.Pq see Sy zfs_scan_mem_lim_fact . 18543ff01b23SMartin MatuskaThis may improve scrub performance if the pool's data is very fragmented. 18553ff01b23SMartin Matuska.It Sy 2 18563ff01b23SMartin MatuskaThe largest mostly-contiguous chunk of found data will be verified first. 18573ff01b23SMartin MatuskaBy deferring scrubbing of small segments, we may later find adjacent data 18583ff01b23SMartin Matuskato coalesce and increase the segment size. 18593ff01b23SMartin Matuska.It Sy 0 18603ff01b23SMartin Matuska.No Use strategy Sy 1 No during normal verification 18613ff01b23SMartin Matuska.No and strategy Sy 2 No while taking a checkpoint . 18623ff01b23SMartin Matuska.El 18633ff01b23SMartin Matuska. 18643ff01b23SMartin Matuska.It Sy zfs_scan_legacy Ns = Ns Sy 0 Ns | Ns 1 Pq int 18653ff01b23SMartin MatuskaIf unset, indicates that scrubs and resilvers will gather metadata in 18663ff01b23SMartin Matuskamemory before issuing sequential I/O. 18673ff01b23SMartin MatuskaOtherwise indicates that the legacy algorithm will be used, 18683ff01b23SMartin Matuskawhere I/O is initiated as soon as it is discovered. 18693ff01b23SMartin MatuskaUnsetting will not affect scrubs or resilvers that are already in progress. 18703ff01b23SMartin Matuska. 1871716fd348SMartin Matuska.It Sy zfs_scan_max_ext_gap Ns = Ns Sy 2097152 Ns B Po 2 MiB Pc Pq int 18723ff01b23SMartin MatuskaSets the largest gap in bytes between scrub/resilver I/O operations 18733ff01b23SMartin Matuskathat will still be considered sequential for sorting purposes. 18743ff01b23SMartin MatuskaChanging this value will not 18753ff01b23SMartin Matuskaaffect scrubs or resilvers that are already in progress. 18763ff01b23SMartin Matuska. 1877be181ee2SMartin Matuska.It Sy zfs_scan_mem_lim_fact Ns = Ns Sy 20 Ns ^-1 Pq uint 18783ff01b23SMartin MatuskaMaximum fraction of RAM used for I/O sorting by sequential scan algorithm. 18793ff01b23SMartin MatuskaThis tunable determines the hard limit for I/O sorting memory usage. 18803ff01b23SMartin MatuskaWhen the hard limit is reached we stop scanning metadata and start issuing 18813ff01b23SMartin Matuskadata verification I/O. 18823ff01b23SMartin MatuskaThis is done until we get below the soft limit. 18833ff01b23SMartin Matuska. 1884be181ee2SMartin Matuska.It Sy zfs_scan_mem_lim_soft_fact Ns = Ns Sy 20 Ns ^-1 Pq uint 18853ff01b23SMartin MatuskaThe fraction of the hard limit used to determined the soft limit for I/O sorting 18863ff01b23SMartin Matuskaby the sequential scan algorithm. 18873ff01b23SMartin MatuskaWhen we cross this limit from below no action is taken. 1888bb2d13b6SMartin MatuskaWhen we cross this limit from above it is because we are issuing verification 1889bb2d13b6SMartin MatuskaI/O. 18903ff01b23SMartin MatuskaIn this case (unless the metadata scan is done) we stop issuing verification I/O 18913ff01b23SMartin Matuskaand start scanning metadata again until we get to the hard limit. 18923ff01b23SMartin Matuska. 18933ff01b23SMartin Matuska.It Sy zfs_scan_strict_mem_lim Ns = Ns Sy 0 Ns | Ns 1 Pq int 18943ff01b23SMartin MatuskaEnforce tight memory limits on pool scans when a sequential scan is in progress. 18953ff01b23SMartin MatuskaWhen disabled, the memory limit may be exceeded by fast disks. 18963ff01b23SMartin Matuska. 18973ff01b23SMartin Matuska.It Sy zfs_scan_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq int 18983ff01b23SMartin MatuskaFreezes a scrub/resilver in progress without actually pausing it. 18993ff01b23SMartin MatuskaIntended for testing/debugging. 19003ff01b23SMartin Matuska. 1901716fd348SMartin Matuska.It Sy zfs_scan_vdev_limit Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq int 19023ff01b23SMartin MatuskaMaximum amount of data that can be concurrently issued at once for scrubs and 19033ff01b23SMartin Matuskaresilvers per leaf device, given in bytes. 19043ff01b23SMartin Matuska. 19053ff01b23SMartin Matuska.It Sy zfs_send_corrupt_data Ns = Ns Sy 0 Ns | Ns 1 Pq int 19063ff01b23SMartin MatuskaAllow sending of corrupt data (ignore read/checksum errors when sending). 19073ff01b23SMartin Matuska. 19083ff01b23SMartin Matuska.It Sy zfs_send_unmodified_spill_blocks Ns = Ns Sy 1 Ns | Ns 0 Pq int 19093ff01b23SMartin MatuskaInclude unmodified spill blocks in the send stream. 19103ff01b23SMartin MatuskaUnder certain circumstances, previous versions of ZFS could incorrectly 19113ff01b23SMartin Matuskaremove the spill block from an existing object. 19123ff01b23SMartin MatuskaIncluding unmodified copies of the spill blocks creates a backwards-compatible 19133ff01b23SMartin Matuskastream which will recreate a spill block if it was incorrectly removed. 19143ff01b23SMartin Matuska. 1915be181ee2SMartin Matuska.It Sy zfs_send_no_prefetch_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq uint 19163ff01b23SMartin MatuskaThe fill fraction of the 19173ff01b23SMartin Matuska.Nm zfs Cm send 19183ff01b23SMartin Matuskainternal queues. 19193ff01b23SMartin MatuskaThe fill fraction controls the timing with which internal threads are woken up. 19203ff01b23SMartin Matuska. 1921be181ee2SMartin Matuska.It Sy zfs_send_no_prefetch_queue_length Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq uint 19223ff01b23SMartin MatuskaThe maximum number of bytes allowed in 19233ff01b23SMartin Matuska.Nm zfs Cm send Ns 's 19243ff01b23SMartin Matuskainternal queues. 19253ff01b23SMartin Matuska. 1926be181ee2SMartin Matuska.It Sy zfs_send_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq uint 19273ff01b23SMartin MatuskaThe fill fraction of the 19283ff01b23SMartin Matuska.Nm zfs Cm send 19293ff01b23SMartin Matuskaprefetch queue. 19303ff01b23SMartin MatuskaThe fill fraction controls the timing with which internal threads are woken up. 19313ff01b23SMartin Matuska. 1932be181ee2SMartin Matuska.It Sy zfs_send_queue_length Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint 19333ff01b23SMartin MatuskaThe maximum number of bytes allowed that will be prefetched by 19343ff01b23SMartin Matuska.Nm zfs Cm send . 19353ff01b23SMartin MatuskaThis value must be at least twice the maximum block size in use. 19363ff01b23SMartin Matuska. 1937be181ee2SMartin Matuska.It Sy zfs_recv_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq uint 19383ff01b23SMartin MatuskaThe fill fraction of the 19393ff01b23SMartin Matuska.Nm zfs Cm receive 19403ff01b23SMartin Matuskaqueue. 19413ff01b23SMartin MatuskaThe fill fraction controls the timing with which internal threads are woken up. 19423ff01b23SMartin Matuska. 1943be181ee2SMartin Matuska.It Sy zfs_recv_queue_length Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint 19443ff01b23SMartin MatuskaThe maximum number of bytes allowed in the 19453ff01b23SMartin Matuska.Nm zfs Cm receive 19463ff01b23SMartin Matuskaqueue. 19473ff01b23SMartin MatuskaThis value must be at least twice the maximum block size in use. 19483ff01b23SMartin Matuska. 1949be181ee2SMartin Matuska.It Sy zfs_recv_write_batch_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq uint 19503ff01b23SMartin MatuskaThe maximum amount of data, in bytes, that 19513ff01b23SMartin Matuska.Nm zfs Cm receive 19523ff01b23SMartin Matuskawill write in one DMU transaction. 19533ff01b23SMartin MatuskaThis is the uncompressed size, even when receiving a compressed send stream. 19543ff01b23SMartin MatuskaThis setting will not reduce the write size below a single block. 19553ff01b23SMartin MatuskaCapped at a maximum of 1956716fd348SMartin Matuska.Sy 32 MiB . 19573ff01b23SMartin Matuska. 1958271171e0SMartin Matuska.It Sy zfs_recv_best_effort_corrective Ns = Ns Sy 0 Pq int 1959271171e0SMartin MatuskaWhen this variable is set to non-zero a corrective receive: 1960271171e0SMartin Matuska.Bl -enum -compact -offset 4n -width "1." 1961271171e0SMartin Matuska.It 1962271171e0SMartin MatuskaDoes not enforce the restriction of source & destination snapshot GUIDs 1963271171e0SMartin Matuskamatching. 1964271171e0SMartin Matuska.It 1965271171e0SMartin MatuskaIf there is an error during healing, the healing receive is not 1966271171e0SMartin Matuskaterminated instead it moves on to the next record. 1967271171e0SMartin Matuska.El 1968271171e0SMartin Matuska. 1969be181ee2SMartin Matuska.It Sy zfs_override_estimate_recordsize Ns = Ns Sy 0 Ns | Ns 1 Pq uint 19703ff01b23SMartin MatuskaSetting this variable overrides the default logic for estimating block 19713ff01b23SMartin Matuskasizes when doing a 19723ff01b23SMartin Matuska.Nm zfs Cm send . 19733ff01b23SMartin MatuskaThe default heuristic is that the average block size 19743ff01b23SMartin Matuskawill be the current recordsize. 19753ff01b23SMartin MatuskaOverride this value if most data in your dataset is not of that size 19763ff01b23SMartin Matuskaand you require accurate zfs send size estimates. 19773ff01b23SMartin Matuska. 1978be181ee2SMartin Matuska.It Sy zfs_sync_pass_deferred_free Ns = Ns Sy 2 Pq uint 19793ff01b23SMartin MatuskaFlushing of data to disk is done in passes. 19803ff01b23SMartin MatuskaDefer frees starting in this pass. 19813ff01b23SMartin Matuska. 1982716fd348SMartin Matuska.It Sy zfs_spa_discard_memory_limit Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq int 19833ff01b23SMartin MatuskaMaximum memory used for prefetching a checkpoint's space map on each 19843ff01b23SMartin Matuskavdev while discarding the checkpoint. 19853ff01b23SMartin Matuska. 1986be181ee2SMartin Matuska.It Sy zfs_special_class_metadata_reserve_pct Ns = Ns Sy 25 Ns % Pq uint 19873ff01b23SMartin MatuskaOnly allow small data blocks to be allocated on the special and dedup vdev 1988bb2d13b6SMartin Matuskatypes when the available free space percentage on these vdevs exceeds this 1989bb2d13b6SMartin Matuskavalue. 19903ff01b23SMartin MatuskaThis ensures reserved space is available for pool metadata as the 19913ff01b23SMartin Matuskaspecial vdevs approach capacity. 19923ff01b23SMartin Matuska. 1993be181ee2SMartin Matuska.It Sy zfs_sync_pass_dont_compress Ns = Ns Sy 8 Pq uint 19943ff01b23SMartin MatuskaStarting in this sync pass, disable compression (including of metadata). 19953ff01b23SMartin MatuskaWith the default setting, in practice, we don't have this many sync passes, 19963ff01b23SMartin Matuskaso this has no effect. 19973ff01b23SMartin Matuska.Pp 19983ff01b23SMartin MatuskaThe original intent was that disabling compression would help the sync passes 19993ff01b23SMartin Matuskato converge. 20003ff01b23SMartin MatuskaHowever, in practice, disabling compression increases 20013ff01b23SMartin Matuskathe average number of sync passes; because when we turn compression off, 20023ff01b23SMartin Matuskamany blocks' size will change, and thus we have to re-allocate 20033ff01b23SMartin Matuska(not overwrite) them. 20043ff01b23SMartin MatuskaIt also increases the number of 2005716fd348SMartin Matuska.Em 128 KiB 20063ff01b23SMartin Matuskaallocations (e.g. for indirect blocks and spacemaps) 20073ff01b23SMartin Matuskabecause these will not be compressed. 20083ff01b23SMartin MatuskaThe 2009716fd348SMartin Matuska.Em 128 KiB 20103ff01b23SMartin Matuskaallocations are especially detrimental to performance 2011bb2d13b6SMartin Matuskaon highly fragmented systems, which may have very few free segments of this 2012bb2d13b6SMartin Matuskasize, 20133ff01b23SMartin Matuskaand may need to load new metaslabs to satisfy these allocations. 20143ff01b23SMartin Matuska. 2015be181ee2SMartin Matuska.It Sy zfs_sync_pass_rewrite Ns = Ns Sy 2 Pq uint 20163ff01b23SMartin MatuskaRewrite new block pointers starting in this pass. 20173ff01b23SMartin Matuska. 20183ff01b23SMartin Matuska.It Sy zfs_sync_taskq_batch_pct Ns = Ns Sy 75 Ns % Pq int 20193ff01b23SMartin MatuskaThis controls the number of threads used by 20203ff01b23SMartin Matuska.Sy dp_sync_taskq . 20213ff01b23SMartin MatuskaThe default value of 20223ff01b23SMartin Matuska.Sy 75% 20233ff01b23SMartin Matuskawill create a maximum of one thread per CPU. 20243ff01b23SMartin Matuska. 2025716fd348SMartin Matuska.It Sy zfs_trim_extent_bytes_max Ns = Ns Sy 134217728 Ns B Po 128 MiB Pc Pq uint 20263ff01b23SMartin MatuskaMaximum size of TRIM command. 2027bb2d13b6SMartin MatuskaLarger ranges will be split into chunks no larger than this value before 2028bb2d13b6SMartin Matuskaissuing. 20293ff01b23SMartin Matuska. 2030716fd348SMartin Matuska.It Sy zfs_trim_extent_bytes_min Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint 20313ff01b23SMartin MatuskaMinimum size of TRIM commands. 20323ff01b23SMartin MatuskaTRIM ranges smaller than this will be skipped, 20333ff01b23SMartin Matuskaunless they're part of a larger range which was chunked. 20343ff01b23SMartin MatuskaThis is done because it's common for these small TRIMs 20353ff01b23SMartin Matuskato negatively impact overall performance. 20363ff01b23SMartin Matuska. 20373ff01b23SMartin Matuska.It Sy zfs_trim_metaslab_skip Ns = Ns Sy 0 Ns | Ns 1 Pq uint 20383ff01b23SMartin MatuskaSkip uninitialized metaslabs during the TRIM process. 2039bb2d13b6SMartin MatuskaThis option is useful for pools constructed from large thinly-provisioned 2040bb2d13b6SMartin Matuskadevices 20413ff01b23SMartin Matuskawhere TRIM operations are slow. 20423ff01b23SMartin MatuskaAs a pool ages, an increasing fraction of the pool's metaslabs 20433ff01b23SMartin Matuskawill be initialized, progressively degrading the usefulness of this option. 20443ff01b23SMartin MatuskaThis setting is stored when starting a manual TRIM and will 20453ff01b23SMartin Matuskapersist for the duration of the requested TRIM. 20463ff01b23SMartin Matuska. 20473ff01b23SMartin Matuska.It Sy zfs_trim_queue_limit Ns = Ns Sy 10 Pq uint 20483ff01b23SMartin MatuskaMaximum number of queued TRIMs outstanding per leaf vdev. 20493ff01b23SMartin MatuskaThe number of concurrent TRIM commands issued to the device is controlled by 20503ff01b23SMartin Matuska.Sy zfs_vdev_trim_min_active No and Sy zfs_vdev_trim_max_active . 20513ff01b23SMartin Matuska. 20523ff01b23SMartin Matuska.It Sy zfs_trim_txg_batch Ns = Ns Sy 32 Pq uint 20533ff01b23SMartin MatuskaThe number of transaction groups' worth of frees which should be aggregated 20543ff01b23SMartin Matuskabefore TRIM operations are issued to the device. 20553ff01b23SMartin MatuskaThis setting represents a trade-off between issuing larger, 20563ff01b23SMartin Matuskamore efficient TRIM operations and the delay 20573ff01b23SMartin Matuskabefore the recently trimmed space is available for use by the device. 20583ff01b23SMartin Matuska.Pp 20593ff01b23SMartin MatuskaIncreasing this value will allow frees to be aggregated for a longer time. 2060bb2d13b6SMartin MatuskaThis will result is larger TRIM operations and potentially increased memory 2061bb2d13b6SMartin Matuskausage. 20623ff01b23SMartin MatuskaDecreasing this value will have the opposite effect. 20633ff01b23SMartin MatuskaThe default of 20643ff01b23SMartin Matuska.Sy 32 20653ff01b23SMartin Matuskawas determined to be a reasonable compromise. 20663ff01b23SMartin Matuska. 2067be181ee2SMartin Matuska.It Sy zfs_txg_history Ns = Ns Sy 0 Pq uint 20683ff01b23SMartin MatuskaHistorical statistics for this many latest TXGs will be available in 20693ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /TXGs . 20703ff01b23SMartin Matuska. 2071be181ee2SMartin Matuska.It Sy zfs_txg_timeout Ns = Ns Sy 5 Ns s Pq uint 2072bb2d13b6SMartin MatuskaFlush dirty data to disk at least every this many seconds (maximum TXG 2073bb2d13b6SMartin Matuskaduration). 20743ff01b23SMartin Matuska. 2075be181ee2SMartin Matuska.It Sy zfs_vdev_aggregate_trim Ns = Ns Sy 0 Ns | Ns 1 Pq uint 2076c03c5b1cSMartin MatuskaAllow TRIM I/O operations to be aggregated. 20773ff01b23SMartin MatuskaThis is normally not helpful because the extents to be trimmed 20783ff01b23SMartin Matuskawill have been already been aggregated by the metaslab. 20793ff01b23SMartin MatuskaThis option is provided for debugging and performance analysis. 20803ff01b23SMartin Matuska. 2081be181ee2SMartin Matuska.It Sy zfs_vdev_aggregation_limit Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq uint 20823ff01b23SMartin MatuskaMax vdev I/O aggregation size. 20833ff01b23SMartin Matuska. 2084be181ee2SMartin Matuska.It Sy zfs_vdev_aggregation_limit_non_rotating Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint 20853ff01b23SMartin MatuskaMax vdev I/O aggregation size for non-rotating media. 20863ff01b23SMartin Matuska. 2087be181ee2SMartin Matuska.It Sy zfs_vdev_cache_bshift Ns = Ns Sy 16 Po 64 KiB Pc Pq uint 20883ff01b23SMartin MatuskaShift size to inflate reads to. 20893ff01b23SMartin Matuska. 2090be181ee2SMartin Matuska.It Sy zfs_vdev_cache_max Ns = Ns Sy 16384 Ns B Po 16 KiB Pc Pq uint 20913ff01b23SMartin MatuskaInflate reads smaller than this value to meet the 20923ff01b23SMartin Matuska.Sy zfs_vdev_cache_bshift 20933ff01b23SMartin Matuskasize 2094716fd348SMartin Matuska.Pq default Sy 64 KiB . 20953ff01b23SMartin Matuska. 2096be181ee2SMartin Matuska.It Sy zfs_vdev_cache_size Ns = Ns Sy 0 Pq uint 20973ff01b23SMartin MatuskaTotal size of the per-disk cache in bytes. 20983ff01b23SMartin Matuska.Pp 20993ff01b23SMartin MatuskaCurrently this feature is disabled, as it has been found to not be helpful 21003ff01b23SMartin Matuskafor performance and in some cases harmful. 21013ff01b23SMartin Matuska. 21023ff01b23SMartin Matuska.It Sy zfs_vdev_mirror_rotating_inc Ns = Ns Sy 0 Pq int 21033ff01b23SMartin MatuskaA number by which the balancing algorithm increments the load calculation for 21043ff01b23SMartin Matuskathe purpose of selecting the least busy mirror member when an I/O operation 21053ff01b23SMartin Matuskaimmediately follows its predecessor on rotational vdevs 21063ff01b23SMartin Matuskafor the purpose of making decisions based on load. 21073ff01b23SMartin Matuska. 21083ff01b23SMartin Matuska.It Sy zfs_vdev_mirror_rotating_seek_inc Ns = Ns Sy 5 Pq int 21093ff01b23SMartin MatuskaA number by which the balancing algorithm increments the load calculation for 21103ff01b23SMartin Matuskathe purpose of selecting the least busy mirror member when an I/O operation 21113ff01b23SMartin Matuskalacks locality as defined by 21123ff01b23SMartin Matuska.Sy zfs_vdev_mirror_rotating_seek_offset . 21133ff01b23SMartin MatuskaOperations within this that are not immediately following the previous operation 21143ff01b23SMartin Matuskaare incremented by half. 21153ff01b23SMartin Matuska. 2116716fd348SMartin Matuska.It Sy zfs_vdev_mirror_rotating_seek_offset Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq int 21173ff01b23SMartin MatuskaThe maximum distance for the last queued I/O operation in which 21183ff01b23SMartin Matuskathe balancing algorithm considers an operation to have locality. 21193ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER . 21203ff01b23SMartin Matuska. 21213ff01b23SMartin Matuska.It Sy zfs_vdev_mirror_non_rotating_inc Ns = Ns Sy 0 Pq int 21223ff01b23SMartin MatuskaA number by which the balancing algorithm increments the load calculation for 21233ff01b23SMartin Matuskathe purpose of selecting the least busy mirror member on non-rotational vdevs 21243ff01b23SMartin Matuskawhen I/O operations do not immediately follow one another. 21253ff01b23SMartin Matuska. 21263ff01b23SMartin Matuska.It Sy zfs_vdev_mirror_non_rotating_seek_inc Ns = Ns Sy 1 Pq int 21273ff01b23SMartin MatuskaA number by which the balancing algorithm increments the load calculation for 2128bb2d13b6SMartin Matuskathe purpose of selecting the least busy mirror member when an I/O operation 2129bb2d13b6SMartin Matuskalacks 21303ff01b23SMartin Matuskalocality as defined by the 21313ff01b23SMartin Matuska.Sy zfs_vdev_mirror_rotating_seek_offset . 21323ff01b23SMartin MatuskaOperations within this that are not immediately following the previous operation 21333ff01b23SMartin Matuskaare incremented by half. 21343ff01b23SMartin Matuska. 2135be181ee2SMartin Matuska.It Sy zfs_vdev_read_gap_limit Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint 21363ff01b23SMartin MatuskaAggregate read I/O operations if the on-disk gap between them is within this 21373ff01b23SMartin Matuskathreshold. 21383ff01b23SMartin Matuska. 2139be181ee2SMartin Matuska.It Sy zfs_vdev_write_gap_limit Ns = Ns Sy 4096 Ns B Po 4 KiB Pc Pq uint 21403ff01b23SMartin MatuskaAggregate write I/O operations if the on-disk gap between them is within this 21413ff01b23SMartin Matuskathreshold. 21423ff01b23SMartin Matuska. 21433ff01b23SMartin Matuska.It Sy zfs_vdev_raidz_impl Ns = Ns Sy fastest Pq string 21443ff01b23SMartin MatuskaSelect the raidz parity implementation to use. 21453ff01b23SMartin Matuska.Pp 21463ff01b23SMartin MatuskaVariants that don't depend on CPU-specific features 21473ff01b23SMartin Matuskamay be selected on module load, as they are supported on all systems. 21483ff01b23SMartin MatuskaThe remaining options may only be set after the module is loaded, 21493ff01b23SMartin Matuskaas they are available only if the implementations are compiled in 21503ff01b23SMartin Matuskaand supported on the running system. 21513ff01b23SMartin Matuska.Pp 21523ff01b23SMartin MatuskaOnce the module is loaded, 21533ff01b23SMartin Matuska.Pa /sys/module/zfs/parameters/zfs_vdev_raidz_impl 21543ff01b23SMartin Matuskawill show the available options, 21553ff01b23SMartin Matuskawith the currently selected one enclosed in square brackets. 21563ff01b23SMartin Matuska.Pp 21573ff01b23SMartin Matuska.TS 21583ff01b23SMartin Matuskalb l l . 21593ff01b23SMartin Matuskafastest selected by built-in benchmark 21603ff01b23SMartin Matuskaoriginal original implementation 21613ff01b23SMartin Matuskascalar scalar implementation 21623ff01b23SMartin Matuskasse2 SSE2 instruction set 64-bit x86 21633ff01b23SMartin Matuskassse3 SSSE3 instruction set 64-bit x86 21643ff01b23SMartin Matuskaavx2 AVX2 instruction set 64-bit x86 21653ff01b23SMartin Matuskaavx512f AVX512F instruction set 64-bit x86 21663ff01b23SMartin Matuskaavx512bw AVX512F & AVX512BW instruction sets 64-bit x86 21673ff01b23SMartin Matuskaaarch64_neon NEON Aarch64/64-bit ARMv8 21683ff01b23SMartin Matuskaaarch64_neonx2 NEON with more unrolling Aarch64/64-bit ARMv8 21693ff01b23SMartin Matuskapowerpc_altivec Altivec PowerPC 21703ff01b23SMartin Matuska.TE 21713ff01b23SMartin Matuska. 21723ff01b23SMartin Matuska.It Sy zfs_vdev_scheduler Pq charp 21733ff01b23SMartin Matuska.Sy DEPRECATED . 21742faf504dSMartin MatuskaPrints warning to kernel log for compatibility. 21753ff01b23SMartin Matuska. 2176be181ee2SMartin Matuska.It Sy zfs_zevent_len_max Ns = Ns Sy 512 Pq uint 21773ff01b23SMartin MatuskaMax event queue length. 21783ff01b23SMartin MatuskaEvents in the queue can be viewed with 21793ff01b23SMartin Matuska.Xr zpool-events 8 . 21803ff01b23SMartin Matuska. 21813ff01b23SMartin Matuska.It Sy zfs_zevent_retain_max Ns = Ns Sy 2000 Pq int 21823ff01b23SMartin MatuskaMaximum recent zevent records to retain for duplicate checking. 21833ff01b23SMartin MatuskaSetting this to 21843ff01b23SMartin Matuska.Sy 0 21853ff01b23SMartin Matuskadisables duplicate detection. 21863ff01b23SMartin Matuska. 21873ff01b23SMartin Matuska.It Sy zfs_zevent_retain_expire_secs Ns = Ns Sy 900 Ns s Po 15 min Pc Pq int 21883ff01b23SMartin MatuskaLifespan for a recent ereport that was retained for duplicate checking. 21893ff01b23SMartin Matuska. 21903ff01b23SMartin Matuska.It Sy zfs_zil_clean_taskq_maxalloc Ns = Ns Sy 1048576 Pq int 21913ff01b23SMartin MatuskaThe maximum number of taskq entries that are allowed to be cached. 21923ff01b23SMartin MatuskaWhen this limit is exceeded transaction records (itxs) 21933ff01b23SMartin Matuskawill be cleaned synchronously. 21943ff01b23SMartin Matuska. 21953ff01b23SMartin Matuska.It Sy zfs_zil_clean_taskq_minalloc Ns = Ns Sy 1024 Pq int 21963ff01b23SMartin MatuskaThe number of taskq entries that are pre-populated when the taskq is first 21973ff01b23SMartin Matuskacreated and are immediately available for use. 21983ff01b23SMartin Matuska. 21993ff01b23SMartin Matuska.It Sy zfs_zil_clean_taskq_nthr_pct Ns = Ns Sy 100 Ns % Pq int 22003ff01b23SMartin MatuskaThis controls the number of threads used by 22013ff01b23SMartin Matuska.Sy dp_zil_clean_taskq . 22023ff01b23SMartin MatuskaThe default value of 22033ff01b23SMartin Matuska.Sy 100% 22043ff01b23SMartin Matuskawill create a maximum of one thread per cpu. 22053ff01b23SMartin Matuska. 2206be181ee2SMartin Matuska.It Sy zil_maxblocksize Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint 22073ff01b23SMartin MatuskaThis sets the maximum block size used by the ZIL. 22083ff01b23SMartin MatuskaOn very fragmented pools, lowering this 2209716fd348SMartin Matuska.Pq typically to Sy 36 KiB 22103ff01b23SMartin Matuskacan improve performance. 22113ff01b23SMartin Matuska. 2212*15f0b8c3SMartin Matuska.It Sy zil_min_commit_timeout Ns = Ns Sy 5000 Pq u64 2213*15f0b8c3SMartin MatuskaThis sets the minimum delay in nanoseconds ZIL care to delay block commit, 2214*15f0b8c3SMartin Matuskawaiting for more records. 2215*15f0b8c3SMartin MatuskaIf ZIL writes are too fast, kernel may not be able sleep for so short interval, 2216*15f0b8c3SMartin Matuskaincreasing log latency above allowed by 2217*15f0b8c3SMartin Matuska.Sy zfs_commit_timeout_pct . 2218*15f0b8c3SMartin Matuska. 22193ff01b23SMartin Matuska.It Sy zil_nocacheflush Ns = Ns Sy 0 Ns | Ns 1 Pq int 22203ff01b23SMartin MatuskaDisable the cache flush commands that are normally sent to disk by 22213ff01b23SMartin Matuskathe ZIL after an LWB write has completed. 22223ff01b23SMartin MatuskaSetting this will cause ZIL corruption on power loss 22233ff01b23SMartin Matuskaif a volatile out-of-order write cache is enabled. 22243ff01b23SMartin Matuska. 22253ff01b23SMartin Matuska.It Sy zil_replay_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int 22263ff01b23SMartin MatuskaDisable intent logging replay. 22273ff01b23SMartin MatuskaCan be disabled for recovery from corrupted ZIL. 22283ff01b23SMartin Matuska. 2229dbd5678dSMartin Matuska.It Sy zil_slog_bulk Ns = Ns Sy 786432 Ns B Po 768 KiB Pc Pq u64 22303ff01b23SMartin MatuskaLimit SLOG write size per commit executed with synchronous priority. 22313ff01b23SMartin MatuskaAny writes above that will be executed with lower (asynchronous) priority 22323ff01b23SMartin Matuskato limit potential SLOG device abuse by single active ZIL writer. 22333ff01b23SMartin Matuska. 2234c03c5b1cSMartin Matuska.It Sy zfs_zil_saxattr Ns = Ns Sy 1 Ns | Ns 0 Pq int 2235c03c5b1cSMartin MatuskaSetting this tunable to zero disables ZIL logging of new 2236c03c5b1cSMartin Matuska.Sy xattr Ns = Ns Sy sa 2237c03c5b1cSMartin Matuskarecords if the 2238c03c5b1cSMartin Matuska.Sy org.openzfs:zilsaxattr 2239c03c5b1cSMartin Matuskafeature is enabled on the pool. 2240c03c5b1cSMartin MatuskaThis would only be necessary to work around bugs in the ZIL logging or replay 2241c03c5b1cSMartin Matuskacode for this record type. 2242c03c5b1cSMartin MatuskaThe tunable has no effect if the feature is disabled. 2243c03c5b1cSMartin Matuska. 2244be181ee2SMartin Matuska.It Sy zfs_embedded_slog_min_ms Ns = Ns Sy 64 Pq uint 22453ff01b23SMartin MatuskaUsually, one metaslab from each normal-class vdev is dedicated for use by 22463ff01b23SMartin Matuskathe ZIL to log synchronous writes. 22473ff01b23SMartin MatuskaHowever, if there are fewer than 22483ff01b23SMartin Matuska.Sy zfs_embedded_slog_min_ms 22493ff01b23SMartin Matuskametaslabs in the vdev, this functionality is disabled. 2250bb2d13b6SMartin MatuskaThis ensures that we don't set aside an unreasonable amount of space for the 2251bb2d13b6SMartin MatuskaZIL. 22523ff01b23SMartin Matuska. 2253be181ee2SMartin Matuska.It Sy zstd_earlyabort_pass Ns = Ns Sy 1 Pq uint 2254e3aa18adSMartin MatuskaWhether heuristic for detection of incompressible data with zstd levels >= 3 2255e3aa18adSMartin Matuskausing LZ4 and zstd-1 passes is enabled. 2256e3aa18adSMartin Matuska. 2257be181ee2SMartin Matuska.It Sy zstd_abort_size Ns = Ns Sy 131072 Pq uint 2258e3aa18adSMartin MatuskaMinimal uncompressed size (inclusive) of a record before the early abort 2259e3aa18adSMartin Matuskaheuristic will be attempted. 2260e3aa18adSMartin Matuska. 22613ff01b23SMartin Matuska.It Sy zio_deadman_log_all Ns = Ns Sy 0 Ns | Ns 1 Pq int 22623ff01b23SMartin MatuskaIf non-zero, the zio deadman will produce debugging messages 22633ff01b23SMartin Matuska.Pq see Sy zfs_dbgmsg_enable 22643ff01b23SMartin Matuskafor all zios, rather than only for leaf zios possessing a vdev. 22653ff01b23SMartin MatuskaThis is meant to be used by developers to gain 22663ff01b23SMartin Matuskadiagnostic information for hang conditions which don't involve a mutex 22673ff01b23SMartin Matuskaor other locking primitive: typically conditions in which a thread in 22683ff01b23SMartin Matuskathe zio pipeline is looping indefinitely. 22693ff01b23SMartin Matuska. 22703ff01b23SMartin Matuska.It Sy zio_slow_io_ms Ns = Ns Sy 30000 Ns ms Po 30 s Pc Pq int 22713ff01b23SMartin MatuskaWhen an I/O operation takes more than this much time to complete, 22723ff01b23SMartin Matuskait's marked as slow. 22733ff01b23SMartin MatuskaEach slow operation causes a delay zevent. 22743ff01b23SMartin MatuskaSlow I/O counters can be seen with 22753ff01b23SMartin Matuska.Nm zpool Cm status Fl s . 22763ff01b23SMartin Matuska. 22773ff01b23SMartin Matuska.It Sy zio_dva_throttle_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 22783ff01b23SMartin MatuskaThrottle block allocations in the I/O pipeline. 22793ff01b23SMartin MatuskaThis allows for dynamic allocation distribution when devices are imbalanced. 22803ff01b23SMartin MatuskaWhen enabled, the maximum number of pending allocations per top-level vdev 22813ff01b23SMartin Matuskais limited by 22823ff01b23SMartin Matuska.Sy zfs_vdev_queue_depth_pct . 22833ff01b23SMartin Matuska. 2284c03c5b1cSMartin Matuska.It Sy zfs_xattr_compat Ns = Ns 0 Ns | Ns 1 Pq int 2285c03c5b1cSMartin MatuskaControl the naming scheme used when setting new xattrs in the user namespace. 2286c03c5b1cSMartin MatuskaIf 2287c03c5b1cSMartin Matuska.Sy 0 2288c03c5b1cSMartin Matuska.Pq the default on Linux , 2289c03c5b1cSMartin Matuskauser namespace xattr names are prefixed with the namespace, to be backwards 2290c03c5b1cSMartin Matuskacompatible with previous versions of ZFS on Linux. 2291c03c5b1cSMartin MatuskaIf 2292c03c5b1cSMartin Matuska.Sy 1 2293c03c5b1cSMartin Matuska.Pq the default on Fx , 2294c03c5b1cSMartin Matuskauser namespace xattr names are not prefixed, to be backwards compatible with 2295c03c5b1cSMartin Matuskaprevious versions of ZFS on illumos and 2296c03c5b1cSMartin Matuska.Fx . 2297c03c5b1cSMartin Matuska.Pp 2298c03c5b1cSMartin MatuskaEither naming scheme can be read on this and future versions of ZFS, regardless 2299c03c5b1cSMartin Matuskaof this tunable, but legacy ZFS on illumos or 2300c03c5b1cSMartin Matuska.Fx 2301c03c5b1cSMartin Matuskaare unable to read user namespace xattrs written in the Linux format, and 2302c03c5b1cSMartin Matuskalegacy versions of ZFS on Linux are unable to read user namespace xattrs written 2303c03c5b1cSMartin Matuskain the legacy ZFS format. 2304c03c5b1cSMartin Matuska.Pp 2305c03c5b1cSMartin MatuskaAn existing xattr with the alternate naming scheme is removed when overwriting 2306c03c5b1cSMartin Matuskathe xattr so as to not accumulate duplicates. 2307c03c5b1cSMartin Matuska. 23083ff01b23SMartin Matuska.It Sy zio_requeue_io_start_cut_in_line Ns = Ns Sy 0 Ns | Ns 1 Pq int 23093ff01b23SMartin MatuskaPrioritize requeued I/O. 23103ff01b23SMartin Matuska. 23113ff01b23SMartin Matuska.It Sy zio_taskq_batch_pct Ns = Ns Sy 80 Ns % Pq uint 23123ff01b23SMartin MatuskaPercentage of online CPUs which will run a worker thread for I/O. 23133ff01b23SMartin MatuskaThese workers are responsible for I/O work such as compression and 23143ff01b23SMartin Matuskachecksum calculations. 23153ff01b23SMartin MatuskaFractional number of CPUs will be rounded down. 23163ff01b23SMartin Matuska.Pp 23173ff01b23SMartin MatuskaThe default value of 23183ff01b23SMartin Matuska.Sy 80% 23193ff01b23SMartin Matuskawas chosen to avoid using all CPUs which can result in 23203ff01b23SMartin Matuskalatency issues and inconsistent application performance, 23213ff01b23SMartin Matuskaespecially when slower compression and/or checksumming is enabled. 23223ff01b23SMartin Matuska. 23233ff01b23SMartin Matuska.It Sy zio_taskq_batch_tpq Ns = Ns Sy 0 Pq uint 23243ff01b23SMartin MatuskaNumber of worker threads per taskq. 23253ff01b23SMartin MatuskaLower values improve I/O ordering and CPU utilization, 23263ff01b23SMartin Matuskawhile higher reduces lock contention. 23273ff01b23SMartin Matuska.Pp 23283ff01b23SMartin MatuskaIf 23293ff01b23SMartin Matuska.Sy 0 , 23303ff01b23SMartin Matuskagenerate a system-dependent value close to 6 threads per taskq. 23313ff01b23SMartin Matuska. 23323ff01b23SMartin Matuska.It Sy zvol_inhibit_dev Ns = Ns Sy 0 Ns | Ns 1 Pq uint 23333ff01b23SMartin MatuskaDo not create zvol device nodes. 23343ff01b23SMartin MatuskaThis may slightly improve startup time on 23353ff01b23SMartin Matuskasystems with a very large number of zvols. 23363ff01b23SMartin Matuska. 23373ff01b23SMartin Matuska.It Sy zvol_major Ns = Ns Sy 230 Pq uint 23383ff01b23SMartin MatuskaMajor number for zvol block devices. 23393ff01b23SMartin Matuska. 2340dbd5678dSMartin Matuska.It Sy zvol_max_discard_blocks Ns = Ns Sy 16384 Pq long 23413ff01b23SMartin MatuskaDiscard (TRIM) operations done on zvols will be done in batches of this 23423ff01b23SMartin Matuskamany blocks, where block size is determined by the 23433ff01b23SMartin Matuska.Sy volblocksize 23443ff01b23SMartin Matuskaproperty of a zvol. 23453ff01b23SMartin Matuska. 2346716fd348SMartin Matuska.It Sy zvol_prefetch_bytes Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint 23473ff01b23SMartin MatuskaWhen adding a zvol to the system, prefetch this many bytes 23483ff01b23SMartin Matuskafrom the start and end of the volume. 23493ff01b23SMartin MatuskaPrefetching these regions of the volume is desirable, 23503ff01b23SMartin Matuskabecause they are likely to be accessed immediately by 23513ff01b23SMartin Matuska.Xr blkid 8 23523ff01b23SMartin Matuskaor the kernel partitioner. 23533ff01b23SMartin Matuska. 23543ff01b23SMartin Matuska.It Sy zvol_request_sync Ns = Ns Sy 0 Ns | Ns 1 Pq uint 23553ff01b23SMartin MatuskaWhen processing I/O requests for a zvol, submit them synchronously. 23563ff01b23SMartin MatuskaThis effectively limits the queue depth to 23573ff01b23SMartin Matuska.Em 1 23583ff01b23SMartin Matuskafor each I/O submitter. 23593ff01b23SMartin MatuskaWhen unset, requests are handled asynchronously by a thread pool. 23603ff01b23SMartin MatuskaThe number of requests which can be handled concurrently is controlled by 23613ff01b23SMartin Matuska.Sy zvol_threads . 23621f1e2261SMartin Matuska.Sy zvol_request_sync 23631f1e2261SMartin Matuskais ignored when running on a kernel that supports block multiqueue 23641f1e2261SMartin Matuska.Pq Li blk-mq . 23653ff01b23SMartin Matuska. 23661f1e2261SMartin Matuska.It Sy zvol_threads Ns = Ns Sy 0 Pq uint 23671f1e2261SMartin MatuskaThe number of system wide threads to use for processing zvol block IOs. 23681f1e2261SMartin MatuskaIf 23691f1e2261SMartin Matuska.Sy 0 23701f1e2261SMartin Matuska(the default) then internally set 23711f1e2261SMartin Matuska.Sy zvol_threads 23721f1e2261SMartin Matuskato the number of CPUs present or 32 (whichever is greater). 23731f1e2261SMartin Matuska. 23741f1e2261SMartin Matuska.It Sy zvol_blk_mq_threads Ns = Ns Sy 0 Pq uint 23751f1e2261SMartin MatuskaThe number of threads per zvol to use for queuing IO requests. 23761f1e2261SMartin MatuskaThis parameter will only appear if your kernel supports 23771f1e2261SMartin Matuska.Li blk-mq 23781f1e2261SMartin Matuskaand is only read and assigned to a zvol at zvol load time. 23791f1e2261SMartin MatuskaIf 23801f1e2261SMartin Matuska.Sy 0 23811f1e2261SMartin Matuska(the default) then internally set 23821f1e2261SMartin Matuska.Sy zvol_blk_mq_threads 23831f1e2261SMartin Matuskato the number of CPUs present. 23841f1e2261SMartin Matuska. 23851f1e2261SMartin Matuska.It Sy zvol_use_blk_mq Ns = Ns Sy 0 Ns | Ns 1 Pq uint 23861f1e2261SMartin MatuskaSet to 23871f1e2261SMartin Matuska.Sy 1 23881f1e2261SMartin Matuskato use the 23891f1e2261SMartin Matuska.Li blk-mq 23901f1e2261SMartin MatuskaAPI for zvols. 23911f1e2261SMartin MatuskaSet to 23921f1e2261SMartin Matuska.Sy 0 23931f1e2261SMartin Matuska(the default) to use the legacy zvol APIs. 23941f1e2261SMartin MatuskaThis setting can give better or worse zvol performance depending on 23951f1e2261SMartin Matuskathe workload. 23961f1e2261SMartin MatuskaThis parameter will only appear if your kernel supports 23971f1e2261SMartin Matuska.Li blk-mq 23981f1e2261SMartin Matuskaand is only read and assigned to a zvol at zvol load time. 23991f1e2261SMartin Matuska. 24001f1e2261SMartin Matuska.It Sy zvol_blk_mq_blocks_per_thread Ns = Ns Sy 8 Pq uint 24011f1e2261SMartin MatuskaIf 24021f1e2261SMartin Matuska.Sy zvol_use_blk_mq 24031f1e2261SMartin Matuskais enabled, then process this number of 24041f1e2261SMartin Matuska.Sy volblocksize Ns -sized blocks per zvol thread. 24051f1e2261SMartin MatuskaThis tunable can be use to favor better performance for zvol reads (lower 24061f1e2261SMartin Matuskavalues) or writes (higher values). 24071f1e2261SMartin MatuskaIf set to 24081f1e2261SMartin Matuska.Sy 0 , 24091f1e2261SMartin Matuskathen the zvol layer will process the maximum number of blocks 24101f1e2261SMartin Matuskaper thread that it can. 24111f1e2261SMartin MatuskaThis parameter will only appear if your kernel supports 24121f1e2261SMartin Matuska.Li blk-mq 24131f1e2261SMartin Matuskaand is only applied at each zvol's load time. 24141f1e2261SMartin Matuska. 24151f1e2261SMartin Matuska.It Sy zvol_blk_mq_queue_depth Ns = Ns Sy 0 Pq uint 24161f1e2261SMartin MatuskaThe queue_depth value for the zvol 24171f1e2261SMartin Matuska.Li blk-mq 24181f1e2261SMartin Matuskainterface. 24191f1e2261SMartin MatuskaThis parameter will only appear if your kernel supports 24201f1e2261SMartin Matuska.Li blk-mq 24211f1e2261SMartin Matuskaand is only applied at each zvol's load time. 24221f1e2261SMartin MatuskaIf 24231f1e2261SMartin Matuska.Sy 0 24241f1e2261SMartin Matuska(the default) then use the kernel's default queue depth. 24251f1e2261SMartin MatuskaValues are clamped to the kernel's 24261f1e2261SMartin Matuska.Dv BLKDEV_MIN_RQ 24271f1e2261SMartin Matuskaand 24281f1e2261SMartin Matuska.Dv BLKDEV_MAX_RQ Ns / Ns Dv BLKDEV_DEFAULT_RQ 24291f1e2261SMartin Matuskalimits. 24303ff01b23SMartin Matuska. 24313ff01b23SMartin Matuska.It Sy zvol_volmode Ns = Ns Sy 1 Pq uint 24323ff01b23SMartin MatuskaDefines zvol block devices behaviour when 24333ff01b23SMartin Matuska.Sy volmode Ns = Ns Sy default : 24343ff01b23SMartin Matuska.Bl -tag -compact -offset 4n -width "a" 24353ff01b23SMartin Matuska.It Sy 1 24363ff01b23SMartin Matuska.No equivalent to Sy full 24373ff01b23SMartin Matuska.It Sy 2 24383ff01b23SMartin Matuska.No equivalent to Sy dev 24393ff01b23SMartin Matuska.It Sy 3 24403ff01b23SMartin Matuska.No equivalent to Sy none 24413ff01b23SMartin Matuska.El 2442dbd5678dSMartin Matuska. 2443dbd5678dSMartin Matuska.It Sy zvol_enforce_quotas Ns = Ns Sy 0 Ns | Ns 1 Pq uint 2444dbd5678dSMartin MatuskaEnable strict ZVOL quota enforcement. 2445dbd5678dSMartin MatuskaThe strict quota enforcement may have a performance impact. 24463ff01b23SMartin Matuska.El 24473ff01b23SMartin Matuska. 24483ff01b23SMartin Matuska.Sh ZFS I/O SCHEDULER 24493ff01b23SMartin MatuskaZFS issues I/O operations to leaf vdevs to satisfy and complete I/O operations. 24503ff01b23SMartin MatuskaThe scheduler determines when and in what order those operations are issued. 24513ff01b23SMartin MatuskaThe scheduler divides operations into five I/O classes, 24523ff01b23SMartin Matuskaprioritized in the following order: sync read, sync write, async read, 24533ff01b23SMartin Matuskaasync write, and scrub/resilver. 24543ff01b23SMartin MatuskaEach queue defines the minimum and maximum number of concurrent operations 24553ff01b23SMartin Matuskathat may be issued to the device. 24563ff01b23SMartin MatuskaIn addition, the device has an aggregate maximum, 24573ff01b23SMartin Matuska.Sy zfs_vdev_max_active . 24583ff01b23SMartin MatuskaNote that the sum of the per-queue minima must not exceed the aggregate maximum. 24593ff01b23SMartin MatuskaIf the sum of the per-queue maxima exceeds the aggregate maximum, 24603ff01b23SMartin Matuskathen the number of active operations may reach 24613ff01b23SMartin Matuska.Sy zfs_vdev_max_active , 24623ff01b23SMartin Matuskain which case no further operations will be issued, 24633ff01b23SMartin Matuskaregardless of whether all per-queue minima have been met. 24643ff01b23SMartin Matuska.Pp 24653ff01b23SMartin MatuskaFor many physical devices, throughput increases with the number of 24663ff01b23SMartin Matuskaconcurrent operations, but latency typically suffers. 24673ff01b23SMartin MatuskaFurthermore, physical devices typically have a limit 24683ff01b23SMartin Matuskaat which more concurrent operations have no 24693ff01b23SMartin Matuskaeffect on throughput or can actually cause it to decrease. 24703ff01b23SMartin Matuska.Pp 24713ff01b23SMartin MatuskaThe scheduler selects the next operation to issue by first looking for an 24723ff01b23SMartin MatuskaI/O class whose minimum has not been satisfied. 24733ff01b23SMartin MatuskaOnce all are satisfied and the aggregate maximum has not been hit, 24743ff01b23SMartin Matuskathe scheduler looks for classes whose maximum has not been satisfied. 24753ff01b23SMartin MatuskaIteration through the I/O classes is done in the order specified above. 24763ff01b23SMartin MatuskaNo further operations are issued 24773ff01b23SMartin Matuskaif the aggregate maximum number of concurrent operations has been hit, 2478bb2d13b6SMartin Matuskaor if there are no operations queued for an I/O class that has not hit its 2479bb2d13b6SMartin Matuskamaximum. 24803ff01b23SMartin MatuskaEvery time an I/O operation is queued or an operation completes, 24813ff01b23SMartin Matuskathe scheduler looks for new operations to issue. 24823ff01b23SMartin Matuska.Pp 24833ff01b23SMartin MatuskaIn general, smaller 24843ff01b23SMartin Matuska.Sy max_active Ns s 24853ff01b23SMartin Matuskawill lead to lower latency of synchronous operations. 24863ff01b23SMartin MatuskaLarger 24873ff01b23SMartin Matuska.Sy max_active Ns s 24883ff01b23SMartin Matuskamay lead to higher overall throughput, depending on underlying storage. 24893ff01b23SMartin Matuska.Pp 24903ff01b23SMartin MatuskaThe ratio of the queues' 24913ff01b23SMartin Matuska.Sy max_active Ns s 24923ff01b23SMartin Matuskadetermines the balance of performance between reads, writes, and scrubs. 24933ff01b23SMartin MatuskaFor example, increasing 24943ff01b23SMartin Matuska.Sy zfs_vdev_scrub_max_active 24953ff01b23SMartin Matuskawill cause the scrub or resilver to complete more quickly, 24963ff01b23SMartin Matuskabut reads and writes to have higher latency and lower throughput. 24973ff01b23SMartin Matuska.Pp 24983ff01b23SMartin MatuskaAll I/O classes have a fixed maximum number of outstanding operations, 24993ff01b23SMartin Matuskaexcept for the async write class. 25003ff01b23SMartin MatuskaAsynchronous writes represent the data that is committed to stable storage 25013ff01b23SMartin Matuskaduring the syncing stage for transaction groups. 25023ff01b23SMartin MatuskaTransaction groups enter the syncing state periodically, 25033ff01b23SMartin Matuskaso the number of queued async writes will quickly burst up 25043ff01b23SMartin Matuskaand then bleed down to zero. 25053ff01b23SMartin MatuskaRather than servicing them as quickly as possible, 25063ff01b23SMartin Matuskathe I/O scheduler changes the maximum number of active async write operations 25073ff01b23SMartin Matuskaaccording to the amount of dirty data in the pool. 25083ff01b23SMartin MatuskaSince both throughput and latency typically increase with the number of 25093ff01b23SMartin Matuskaconcurrent operations issued to physical devices, reducing the 2510bb2d13b6SMartin Matuskaburstiness in the number of simultaneous operations also stabilizes the 2511bb2d13b6SMartin Matuskaresponse time of operations from other queues, in particular synchronous ones. 25123ff01b23SMartin MatuskaIn broad strokes, the I/O scheduler will issue more concurrent operations 2513bb2d13b6SMartin Matuskafrom the async write queue as there is more dirty data in the pool. 25143ff01b23SMartin Matuska. 25153ff01b23SMartin Matuska.Ss Async Writes 25163ff01b23SMartin MatuskaThe number of concurrent operations issued for the async write I/O class 25173ff01b23SMartin Matuskafollows a piece-wise linear function defined by a few adjustable points: 25183ff01b23SMartin Matuska.Bd -literal 25193ff01b23SMartin Matuska | o---------| <-- \fBzfs_vdev_async_write_max_active\fP 25203ff01b23SMartin Matuska ^ | /^ | 25213ff01b23SMartin Matuska | | / | | 25223ff01b23SMartin Matuskaactive | / | | 25233ff01b23SMartin Matuska I/O | / | | 25243ff01b23SMartin Matuskacount | / | | 25253ff01b23SMartin Matuska | / | | 25263ff01b23SMartin Matuska |-------o | | <-- \fBzfs_vdev_async_write_min_active\fP 25273ff01b23SMartin Matuska 0|_______^______|_________| 25283ff01b23SMartin Matuska 0% | | 100% of \fBzfs_dirty_data_max\fP 25293ff01b23SMartin Matuska | | 25303ff01b23SMartin Matuska | `-- \fBzfs_vdev_async_write_active_max_dirty_percent\fP 25313ff01b23SMartin Matuska `--------- \fBzfs_vdev_async_write_active_min_dirty_percent\fP 25323ff01b23SMartin Matuska.Ed 25333ff01b23SMartin Matuska.Pp 25343ff01b23SMartin MatuskaUntil the amount of dirty data exceeds a minimum percentage of the dirty 25353ff01b23SMartin Matuskadata allowed in the pool, the I/O scheduler will limit the number of 25363ff01b23SMartin Matuskaconcurrent operations to the minimum. 25373ff01b23SMartin MatuskaAs that threshold is crossed, the number of concurrent operations issued 25383ff01b23SMartin Matuskaincreases linearly to the maximum at the specified maximum percentage 25393ff01b23SMartin Matuskaof the dirty data allowed in the pool. 25403ff01b23SMartin Matuska.Pp 25413ff01b23SMartin MatuskaIdeally, the amount of dirty data on a busy pool will stay in the sloped 25423ff01b23SMartin Matuskapart of the function between 25433ff01b23SMartin Matuska.Sy zfs_vdev_async_write_active_min_dirty_percent 25443ff01b23SMartin Matuskaand 25453ff01b23SMartin Matuska.Sy zfs_vdev_async_write_active_max_dirty_percent . 25463ff01b23SMartin MatuskaIf it exceeds the maximum percentage, 25473ff01b23SMartin Matuskathis indicates that the rate of incoming data is 25483ff01b23SMartin Matuskagreater than the rate that the backend storage can handle. 25493ff01b23SMartin MatuskaIn this case, we must further throttle incoming writes, 25503ff01b23SMartin Matuskaas described in the next section. 25513ff01b23SMartin Matuska. 25523ff01b23SMartin Matuska.Sh ZFS TRANSACTION DELAY 25533ff01b23SMartin MatuskaWe delay transactions when we've determined that the backend storage 25543ff01b23SMartin Matuskaisn't able to accommodate the rate of incoming writes. 25553ff01b23SMartin Matuska.Pp 25563ff01b23SMartin MatuskaIf there is already a transaction waiting, we delay relative to when 25573ff01b23SMartin Matuskathat transaction will finish waiting. 25583ff01b23SMartin MatuskaThis way the calculated delay time 25593ff01b23SMartin Matuskais independent of the number of threads concurrently executing transactions. 25603ff01b23SMartin Matuska.Pp 25613ff01b23SMartin MatuskaIf we are the only waiter, wait relative to when the transaction started, 25623ff01b23SMartin Matuskarather than the current time. 25633ff01b23SMartin MatuskaThis credits the transaction for "time already served", 25643ff01b23SMartin Matuskae.g. reading indirect blocks. 25653ff01b23SMartin Matuska.Pp 25663ff01b23SMartin MatuskaThe minimum time for a transaction to take is calculated as 2567e92ffd9bSMartin Matuska.D1 min_time = min( Ns Sy zfs_delay_scale No \(mu Po Sy dirty No \- Sy min Pc / Po Sy max No \- Sy dirty Pc , 100ms) 25683ff01b23SMartin Matuska.Pp 25693ff01b23SMartin MatuskaThe delay has two degrees of freedom that can be adjusted via tunables. 25703ff01b23SMartin MatuskaThe percentage of dirty data at which we start to delay is defined by 25713ff01b23SMartin Matuska.Sy zfs_delay_min_dirty_percent . 25723ff01b23SMartin MatuskaThis should typically be at or above 25733ff01b23SMartin Matuska.Sy zfs_vdev_async_write_active_max_dirty_percent , 25743ff01b23SMartin Matuskaso that we only start to delay after writing at full speed 25753ff01b23SMartin Matuskahas failed to keep up with the incoming write rate. 25763ff01b23SMartin MatuskaThe scale of the curve is defined by 25773ff01b23SMartin Matuska.Sy zfs_delay_scale . 2578bb2d13b6SMartin MatuskaRoughly speaking, this variable determines the amount of delay at the midpoint 2579bb2d13b6SMartin Matuskaof the curve. 25803ff01b23SMartin Matuska.Bd -literal 25813ff01b23SMartin Matuskadelay 25823ff01b23SMartin Matuska 10ms +-------------------------------------------------------------*+ 25833ff01b23SMartin Matuska | *| 25843ff01b23SMartin Matuska 9ms + *+ 25853ff01b23SMartin Matuska | *| 25863ff01b23SMartin Matuska 8ms + *+ 25873ff01b23SMartin Matuska | * | 25883ff01b23SMartin Matuska 7ms + * + 25893ff01b23SMartin Matuska | * | 25903ff01b23SMartin Matuska 6ms + * + 25913ff01b23SMartin Matuska | * | 25923ff01b23SMartin Matuska 5ms + * + 25933ff01b23SMartin Matuska | * | 25943ff01b23SMartin Matuska 4ms + * + 25953ff01b23SMartin Matuska | * | 25963ff01b23SMartin Matuska 3ms + * + 25973ff01b23SMartin Matuska | * | 25983ff01b23SMartin Matuska 2ms + (midpoint) * + 25993ff01b23SMartin Matuska | | ** | 26003ff01b23SMartin Matuska 1ms + v *** + 26013ff01b23SMartin Matuska | \fBzfs_delay_scale\fP ----------> ******** | 26023ff01b23SMartin Matuska 0 +-------------------------------------*********----------------+ 26033ff01b23SMartin Matuska 0% <- \fBzfs_dirty_data_max\fP -> 100% 26043ff01b23SMartin Matuska.Ed 26053ff01b23SMartin Matuska.Pp 26063ff01b23SMartin MatuskaNote, that since the delay is added to the outstanding time remaining on the 26073ff01b23SMartin Matuskamost recent transaction it's effectively the inverse of IOPS. 26083ff01b23SMartin MatuskaHere, the midpoint of 26093ff01b23SMartin Matuska.Em 500 us 26103ff01b23SMartin Matuskatranslates to 26113ff01b23SMartin Matuska.Em 2000 IOPS . 26123ff01b23SMartin MatuskaThe shape of the curve 26133ff01b23SMartin Matuskawas chosen such that small changes in the amount of accumulated dirty data 26143ff01b23SMartin Matuskain the first three quarters of the curve yield relatively small differences 26153ff01b23SMartin Matuskain the amount of delay. 26163ff01b23SMartin Matuska.Pp 26173ff01b23SMartin MatuskaThe effects can be easier to understand when the amount of delay is 26183ff01b23SMartin Matuskarepresented on a logarithmic scale: 26193ff01b23SMartin Matuska.Bd -literal 26203ff01b23SMartin Matuskadelay 26213ff01b23SMartin Matuska100ms +-------------------------------------------------------------++ 26223ff01b23SMartin Matuska + + 26233ff01b23SMartin Matuska | | 26243ff01b23SMartin Matuska + *+ 26253ff01b23SMartin Matuska 10ms + *+ 26263ff01b23SMartin Matuska + ** + 26273ff01b23SMartin Matuska | (midpoint) ** | 26283ff01b23SMartin Matuska + | ** + 26293ff01b23SMartin Matuska 1ms + v **** + 26303ff01b23SMartin Matuska + \fBzfs_delay_scale\fP ----------> ***** + 26313ff01b23SMartin Matuska | **** | 26323ff01b23SMartin Matuska + **** + 26333ff01b23SMartin Matuska100us + ** + 26343ff01b23SMartin Matuska + * + 26353ff01b23SMartin Matuska | * | 26363ff01b23SMartin Matuska + * + 26373ff01b23SMartin Matuska 10us + * + 26383ff01b23SMartin Matuska + + 26393ff01b23SMartin Matuska | | 26403ff01b23SMartin Matuska + + 26413ff01b23SMartin Matuska +--------------------------------------------------------------+ 26423ff01b23SMartin Matuska 0% <- \fBzfs_dirty_data_max\fP -> 100% 26433ff01b23SMartin Matuska.Ed 26443ff01b23SMartin Matuska.Pp 26453ff01b23SMartin MatuskaNote here that only as the amount of dirty data approaches its limit does 26463ff01b23SMartin Matuskathe delay start to increase rapidly. 26473ff01b23SMartin MatuskaThe goal of a properly tuned system should be to keep the amount of dirty data 26483ff01b23SMartin Matuskaout of that range by first ensuring that the appropriate limits are set 26493ff01b23SMartin Matuskafor the I/O scheduler to reach optimal throughput on the back-end storage, 26503ff01b23SMartin Matuskaand then by changing the value of 26513ff01b23SMartin Matuska.Sy zfs_delay_scale 26523ff01b23SMartin Matuskato increase the steepness of the curve. 2653