xref: /freebsd/sys/contrib/openzfs/man/man4/zfs.4 (revision 15f0b8c309dea1dcb14d3e374686576ff68ac43f)
13ff01b23SMartin Matuska.\"
23ff01b23SMartin Matuska.\" Copyright (c) 2013 by Turbo Fredriksson <turbo@bayour.com>. All rights reserved.
33ff01b23SMartin Matuska.\" Copyright (c) 2019, 2021 by Delphix. All rights reserved.
43ff01b23SMartin Matuska.\" Copyright (c) 2019 Datto Inc.
53ff01b23SMartin Matuska.\" The contents of this file are subject to the terms of the Common Development
63ff01b23SMartin Matuska.\" and Distribution License (the "License").  You may not use this file except
73ff01b23SMartin Matuska.\" in compliance with the License. You can obtain a copy of the license at
8271171e0SMartin Matuska.\" usr/src/OPENSOLARIS.LICENSE or https://opensource.org/licenses/CDDL-1.0.
93ff01b23SMartin Matuska.\"
103ff01b23SMartin Matuska.\" See the License for the specific language governing permissions and
113ff01b23SMartin Matuska.\" limitations under the License. When distributing Covered Code, include this
123ff01b23SMartin Matuska.\" CDDL HEADER in each file and include the License file at
133ff01b23SMartin Matuska.\" usr/src/OPENSOLARIS.LICENSE.  If applicable, add the following below this
143ff01b23SMartin Matuska.\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your
153ff01b23SMartin Matuska.\" own identifying information:
163ff01b23SMartin Matuska.\" Portions Copyright [yyyy] [name of copyright owner]
173ff01b23SMartin Matuska.\"
18*15f0b8c3SMartin Matuska.Dd January 10, 2023
193ff01b23SMartin Matuska.Dt ZFS 4
203ff01b23SMartin Matuska.Os
213ff01b23SMartin Matuska.
223ff01b23SMartin Matuska.Sh NAME
233ff01b23SMartin Matuska.Nm zfs
243ff01b23SMartin Matuska.Nd tuning of the ZFS kernel module
253ff01b23SMartin Matuska.
263ff01b23SMartin Matuska.Sh DESCRIPTION
273ff01b23SMartin MatuskaThe ZFS module supports these parameters:
283ff01b23SMartin Matuska.Bl -tag -width Ds
29dbd5678dSMartin Matuska.It Sy dbuf_cache_max_bytes Ns = Ns Sy UINT64_MAX Ns B Pq u64
303ff01b23SMartin MatuskaMaximum size in bytes of the dbuf cache.
313ff01b23SMartin MatuskaThe target size is determined by the MIN versus
323ff01b23SMartin Matuska.No 1/2^ Ns Sy dbuf_cache_shift Pq 1/32nd
333ff01b23SMartin Matuskaof the target ARC size.
343ff01b23SMartin MatuskaThe behavior of the dbuf cache and its associated settings
353ff01b23SMartin Matuskacan be observed via the
363ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs/dbufstats
373ff01b23SMartin Matuskakstat.
383ff01b23SMartin Matuska.
39dbd5678dSMartin Matuska.It Sy dbuf_metadata_cache_max_bytes Ns = Ns Sy UINT64_MAX Ns B Pq u64
403ff01b23SMartin MatuskaMaximum size in bytes of the metadata dbuf cache.
413ff01b23SMartin MatuskaThe target size is determined by the MIN versus
423ff01b23SMartin Matuska.No 1/2^ Ns Sy dbuf_metadata_cache_shift Pq 1/64th
433ff01b23SMartin Matuskaof the target ARC size.
443ff01b23SMartin MatuskaThe behavior of the metadata dbuf cache and its associated settings
453ff01b23SMartin Matuskacan be observed via the
463ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs/dbufstats
473ff01b23SMartin Matuskakstat.
483ff01b23SMartin Matuska.
493ff01b23SMartin Matuska.It Sy dbuf_cache_hiwater_pct Ns = Ns Sy 10 Ns % Pq uint
503ff01b23SMartin MatuskaThe percentage over
513ff01b23SMartin Matuska.Sy dbuf_cache_max_bytes
523ff01b23SMartin Matuskawhen dbufs must be evicted directly.
533ff01b23SMartin Matuska.
543ff01b23SMartin Matuska.It Sy dbuf_cache_lowater_pct Ns = Ns Sy 10 Ns % Pq uint
553ff01b23SMartin MatuskaThe percentage below
563ff01b23SMartin Matuska.Sy dbuf_cache_max_bytes
573ff01b23SMartin Matuskawhen the evict thread stops evicting dbufs.
583ff01b23SMartin Matuska.
59be181ee2SMartin Matuska.It Sy dbuf_cache_shift Ns = Ns Sy 5 Pq uint
603ff01b23SMartin MatuskaSet the size of the dbuf cache
613ff01b23SMartin Matuska.Pq Sy dbuf_cache_max_bytes
623ff01b23SMartin Matuskato a log2 fraction of the target ARC size.
633ff01b23SMartin Matuska.
64be181ee2SMartin Matuska.It Sy dbuf_metadata_cache_shift Ns = Ns Sy 6 Pq uint
653ff01b23SMartin MatuskaSet the size of the dbuf metadata cache
663ff01b23SMartin Matuska.Pq Sy dbuf_metadata_cache_max_bytes
673ff01b23SMartin Matuskato a log2 fraction of the target ARC size.
683ff01b23SMartin Matuska.
69be181ee2SMartin Matuska.It Sy dbuf_mutex_cache_shift Ns = Ns Sy 0 Pq uint
70be181ee2SMartin MatuskaSet the size of the mutex array for the dbuf cache.
71be181ee2SMartin MatuskaWhen set to
72be181ee2SMartin Matuska.Sy 0
73be181ee2SMartin Matuskathe array is dynamically sized based on total system memory.
74be181ee2SMartin Matuska.
75be181ee2SMartin Matuska.It Sy dmu_object_alloc_chunk_shift Ns = Ns Sy 7 Po 128 Pc Pq uint
763ff01b23SMartin Matuskadnode slots allocated in a single operation as a power of 2.
773ff01b23SMartin MatuskaThe default value minimizes lock contention for the bulk operation performed.
783ff01b23SMartin Matuska.
79be181ee2SMartin Matuska.It Sy dmu_prefetch_max Ns = Ns Sy 134217728 Ns B Po 128 MiB Pc Pq uint
803ff01b23SMartin MatuskaLimit the amount we can prefetch with one call to this amount in bytes.
813ff01b23SMartin MatuskaThis helps to limit the amount of memory that can be used by prefetching.
823ff01b23SMartin Matuska.
833ff01b23SMartin Matuska.It Sy ignore_hole_birth Pq int
843ff01b23SMartin MatuskaAlias for
853ff01b23SMartin Matuska.Sy send_holes_without_birth_time .
863ff01b23SMartin Matuska.
873ff01b23SMartin Matuska.It Sy l2arc_feed_again Ns = Ns Sy 1 Ns | Ns 0 Pq int
883ff01b23SMartin MatuskaTurbo L2ARC warm-up.
893ff01b23SMartin MatuskaWhen the L2ARC is cold the fill interval will be set as fast as possible.
903ff01b23SMartin Matuska.
91dbd5678dSMartin Matuska.It Sy l2arc_feed_min_ms Ns = Ns Sy 200 Pq u64
923ff01b23SMartin MatuskaMin feed interval in milliseconds.
933ff01b23SMartin MatuskaRequires
943ff01b23SMartin Matuska.Sy l2arc_feed_again Ns = Ns Ar 1
953ff01b23SMartin Matuskaand only applicable in related situations.
963ff01b23SMartin Matuska.
97dbd5678dSMartin Matuska.It Sy l2arc_feed_secs Ns = Ns Sy 1 Pq u64
983ff01b23SMartin MatuskaSeconds between L2ARC writing.
993ff01b23SMartin Matuska.
100dbd5678dSMartin Matuska.It Sy l2arc_headroom Ns = Ns Sy 2 Pq u64
1013ff01b23SMartin MatuskaHow far through the ARC lists to search for L2ARC cacheable content,
1023ff01b23SMartin Matuskaexpressed as a multiplier of
1033ff01b23SMartin Matuska.Sy l2arc_write_max .
1043ff01b23SMartin MatuskaARC persistence across reboots can be achieved with persistent L2ARC
1053ff01b23SMartin Matuskaby setting this parameter to
1063ff01b23SMartin Matuska.Sy 0 ,
1073ff01b23SMartin Matuskaallowing the full length of ARC lists to be searched for cacheable content.
1083ff01b23SMartin Matuska.
109dbd5678dSMartin Matuska.It Sy l2arc_headroom_boost Ns = Ns Sy 200 Ns % Pq u64
1103ff01b23SMartin MatuskaScales
1113ff01b23SMartin Matuska.Sy l2arc_headroom
1123ff01b23SMartin Matuskaby this percentage when L2ARC contents are being successfully compressed
1133ff01b23SMartin Matuskabefore writing.
1143ff01b23SMartin MatuskaA value of
1153ff01b23SMartin Matuska.Sy 100
1163ff01b23SMartin Matuskadisables this feature.
1173ff01b23SMartin Matuska.
118dae17134SMartin Matuska.It Sy l2arc_exclude_special Ns = Ns Sy 0 Ns | Ns 1 Pq int
119e92ffd9bSMartin MatuskaControls whether buffers present on special vdevs are eligible for caching
120dae17134SMartin Matuskainto L2ARC.
121dae17134SMartin MatuskaIf set to 1, exclude dbufs on special vdevs from being cached to L2ARC.
122dae17134SMartin Matuska.
1233ff01b23SMartin Matuska.It Sy l2arc_mfuonly Ns = Ns Sy 0 Ns | Ns 1 Pq  int
1243ff01b23SMartin MatuskaControls whether only MFU metadata and data are cached from ARC into L2ARC.
1253ff01b23SMartin MatuskaThis may be desired to avoid wasting space on L2ARC when reading/writing large
1263ff01b23SMartin Matuskaamounts of data that are not expected to be accessed more than once.
1273ff01b23SMartin Matuska.Pp
1283ff01b23SMartin MatuskaThe default is off,
1293ff01b23SMartin Matuskameaning both MRU and MFU data and metadata are cached.
1303ff01b23SMartin MatuskaWhen turning off this feature, some MRU buffers will still be present
1313ff01b23SMartin Matuskain ARC and eventually cached on L2ARC.
1323ff01b23SMartin Matuska.No If Sy l2arc_noprefetch Ns = Ns Sy 0 ,
1333ff01b23SMartin Matuskasome prefetched buffers will be cached to L2ARC, and those might later
1343ff01b23SMartin Matuskatransition to MRU, in which case the
1353ff01b23SMartin Matuska.Sy l2arc_mru_asize No arcstat will not be Sy 0 .
1363ff01b23SMartin Matuska.Pp
1373ff01b23SMartin MatuskaRegardless of
1383ff01b23SMartin Matuska.Sy l2arc_noprefetch ,
1393ff01b23SMartin Matuskasome MFU buffers might be evicted from ARC,
1403ff01b23SMartin Matuskaaccessed later on as prefetches and transition to MRU as prefetches.
1413ff01b23SMartin MatuskaIf accessed again they are counted as MRU and the
1423ff01b23SMartin Matuska.Sy l2arc_mru_asize No arcstat will not be Sy 0 .
1433ff01b23SMartin Matuska.Pp
1443ff01b23SMartin MatuskaThe ARC status of L2ARC buffers when they were first cached in
1453ff01b23SMartin MatuskaL2ARC can be seen in the
1463ff01b23SMartin Matuska.Sy l2arc_mru_asize , Sy l2arc_mfu_asize , No and Sy l2arc_prefetch_asize
1473ff01b23SMartin Matuskaarcstats when importing the pool or onlining a cache
1483ff01b23SMartin Matuskadevice if persistent L2ARC is enabled.
1493ff01b23SMartin Matuska.Pp
1503ff01b23SMartin MatuskaThe
1513ff01b23SMartin Matuska.Sy evict_l2_eligible_mru
1523ff01b23SMartin Matuskaarcstat does not take into account if this option is enabled as the information
1533ff01b23SMartin Matuskaprovided by the
1543ff01b23SMartin Matuska.Sy evict_l2_eligible_m[rf]u
1553ff01b23SMartin Matuskaarcstats can be used to decide if toggling this option is appropriate
1563ff01b23SMartin Matuskafor the current workload.
1573ff01b23SMartin Matuska.
158be181ee2SMartin Matuska.It Sy l2arc_meta_percent Ns = Ns Sy 33 Ns % Pq uint
1593ff01b23SMartin MatuskaPercent of ARC size allowed for L2ARC-only headers.
1603ff01b23SMartin MatuskaSince L2ARC buffers are not evicted on memory pressure,
1613ff01b23SMartin Matuskatoo many headers on a system with an irrationally large L2ARC
1623ff01b23SMartin Matuskacan render it slow or unusable.
1633ff01b23SMartin MatuskaThis parameter limits L2ARC writes and rebuilds to achieve the target.
1643ff01b23SMartin Matuska.
165dbd5678dSMartin Matuska.It Sy l2arc_trim_ahead Ns = Ns Sy 0 Ns % Pq u64
1663ff01b23SMartin MatuskaTrims ahead of the current write size
1673ff01b23SMartin Matuska.Pq Sy l2arc_write_max
1683ff01b23SMartin Matuskaon L2ARC devices by this percentage of write size if we have filled the device.
1693ff01b23SMartin MatuskaIf set to
1703ff01b23SMartin Matuska.Sy 100
1713ff01b23SMartin Matuskawe TRIM twice the space required to accommodate upcoming writes.
1723ff01b23SMartin MatuskaA minimum of
173716fd348SMartin Matuska.Sy 64 MiB
1743ff01b23SMartin Matuskawill be trimmed.
1753ff01b23SMartin MatuskaIt also enables TRIM of the whole L2ARC device upon creation
1763ff01b23SMartin Matuskaor addition to an existing pool or if the header of the device is
1773ff01b23SMartin Matuskainvalid upon importing a pool or onlining a cache device.
1783ff01b23SMartin MatuskaA value of
1793ff01b23SMartin Matuska.Sy 0
1803ff01b23SMartin Matuskadisables TRIM on L2ARC altogether and is the default as it can put significant
1813ff01b23SMartin Matuskastress on the underlying storage devices.
1823ff01b23SMartin MatuskaThis will vary depending of how well the specific device handles these commands.
1833ff01b23SMartin Matuska.
1843ff01b23SMartin Matuska.It Sy l2arc_noprefetch Ns = Ns Sy 1 Ns | Ns 0 Pq int
1853ff01b23SMartin MatuskaDo not write buffers to L2ARC if they were prefetched but not used by
1863ff01b23SMartin Matuskaapplications.
1873ff01b23SMartin MatuskaIn case there are prefetched buffers in L2ARC and this option
1883ff01b23SMartin Matuskais later set, we do not read the prefetched buffers from L2ARC.
1893ff01b23SMartin MatuskaUnsetting this option is useful for caching sequential reads from the
1903ff01b23SMartin Matuskadisks to L2ARC and serve those reads from L2ARC later on.
1913ff01b23SMartin MatuskaThis may be beneficial in case the L2ARC device is significantly faster
1923ff01b23SMartin Matuskain sequential reads than the disks of the pool.
1933ff01b23SMartin Matuska.Pp
1943ff01b23SMartin MatuskaUse
1953ff01b23SMartin Matuska.Sy 1
1963ff01b23SMartin Matuskato disable and
1973ff01b23SMartin Matuska.Sy 0
1983ff01b23SMartin Matuskato enable caching/reading prefetches to/from L2ARC.
1993ff01b23SMartin Matuska.
2003ff01b23SMartin Matuska.It Sy l2arc_norw Ns = Ns Sy 0 Ns | Ns 1 Pq int
2013ff01b23SMartin MatuskaNo reads during writes.
2023ff01b23SMartin Matuska.
203dbd5678dSMartin Matuska.It Sy l2arc_write_boost Ns = Ns Sy 8388608 Ns B Po 8 MiB Pc Pq u64
2043ff01b23SMartin MatuskaCold L2ARC devices will have
2053ff01b23SMartin Matuska.Sy l2arc_write_max
2063ff01b23SMartin Matuskaincreased by this amount while they remain cold.
2073ff01b23SMartin Matuska.
208dbd5678dSMartin Matuska.It Sy l2arc_write_max Ns = Ns Sy 8388608 Ns B Po 8 MiB Pc Pq u64
2093ff01b23SMartin MatuskaMax write bytes per interval.
2103ff01b23SMartin Matuska.
2113ff01b23SMartin Matuska.It Sy l2arc_rebuild_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
2123ff01b23SMartin MatuskaRebuild the L2ARC when importing a pool (persistent L2ARC).
2133ff01b23SMartin MatuskaThis can be disabled if there are problems importing a pool
2143ff01b23SMartin Matuskaor attaching an L2ARC device (e.g. the L2ARC device is slow
2153ff01b23SMartin Matuskain reading stored log metadata, or the metadata
2163ff01b23SMartin Matuskahas become somehow fragmented/unusable).
2173ff01b23SMartin Matuska.
218dbd5678dSMartin Matuska.It Sy l2arc_rebuild_blocks_min_l2size Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq u64
2193ff01b23SMartin MatuskaMininum size of an L2ARC device required in order to write log blocks in it.
2203ff01b23SMartin MatuskaThe log blocks are used upon importing the pool to rebuild the persistent L2ARC.
2213ff01b23SMartin Matuska.Pp
222716fd348SMartin MatuskaFor L2ARC devices less than 1 GiB, the amount of data
2233ff01b23SMartin Matuska.Fn l2arc_evict
2243ff01b23SMartin Matuskaevicts is significant compared to the amount of restored L2ARC data.
2253ff01b23SMartin MatuskaIn this case, do not write log blocks in L2ARC in order not to waste space.
2263ff01b23SMartin Matuska.
227dbd5678dSMartin Matuska.It Sy metaslab_aliquot Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64
2283ff01b23SMartin MatuskaMetaslab granularity, in bytes.
2293ff01b23SMartin MatuskaThis is roughly similar to what would be referred to as the "stripe size"
2303ff01b23SMartin Matuskain traditional RAID arrays.
231716fd348SMartin MatuskaIn normal operation, ZFS will try to write this amount of data to each disk
232716fd348SMartin Matuskabefore moving on to the next top-level vdev.
2333ff01b23SMartin Matuska.
2343ff01b23SMartin Matuska.It Sy metaslab_bias_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
2353ff01b23SMartin MatuskaEnable metaslab group biasing based on their vdevs' over- or under-utilization
2363ff01b23SMartin Matuskarelative to the pool.
2373ff01b23SMartin Matuska.
238dbd5678dSMartin Matuska.It Sy metaslab_force_ganging Ns = Ns Sy 16777217 Ns B Po 16 MiB + 1 B Pc Pq u64
2393ff01b23SMartin MatuskaMake some blocks above a certain size be gang blocks.
2403ff01b23SMartin MatuskaThis option is used by the test suite to facilitate testing.
2413ff01b23SMartin Matuska.
242*15f0b8c3SMartin Matuska.It Sy zfs_default_bs Ns = Ns Sy 9 Po 512 B Pc Pq int
243*15f0b8c3SMartin MatuskaDefault dnode block size as a power of 2.
244*15f0b8c3SMartin Matuska.
245*15f0b8c3SMartin Matuska.It Sy zfs_default_ibs Ns = Ns Sy 17 Po 128 KiB Pc Pq int
246*15f0b8c3SMartin MatuskaDefault dnode indirect block size as a power of 2.
247*15f0b8c3SMartin Matuska.
248dbd5678dSMartin Matuska.It Sy zfs_history_output_max Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64
2493ff01b23SMartin MatuskaWhen attempting to log an output nvlist of an ioctl in the on-disk history,
2503ff01b23SMartin Matuskathe output will not be stored if it is larger than this size (in bytes).
2513ff01b23SMartin MatuskaThis must be less than
252716fd348SMartin Matuska.Sy DMU_MAX_ACCESS Pq 64 MiB .
2533ff01b23SMartin MatuskaThis applies primarily to
2543ff01b23SMartin Matuska.Fn zfs_ioc_channel_program Pq cf. Xr zfs-program 8 .
2553ff01b23SMartin Matuska.
2563ff01b23SMartin Matuska.It Sy zfs_keep_log_spacemaps_at_export Ns = Ns Sy 0 Ns | Ns 1 Pq int
2573ff01b23SMartin MatuskaPrevent log spacemaps from being destroyed during pool exports and destroys.
2583ff01b23SMartin Matuska.
2593ff01b23SMartin Matuska.It Sy zfs_metaslab_segment_weight_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
2603ff01b23SMartin MatuskaEnable/disable segment-based metaslab selection.
2613ff01b23SMartin Matuska.
2623ff01b23SMartin Matuska.It Sy zfs_metaslab_switch_threshold Ns = Ns Sy 2 Pq int
2633ff01b23SMartin MatuskaWhen using segment-based metaslab selection, continue allocating
2643ff01b23SMartin Matuskafrom the active metaslab until this option's
2653ff01b23SMartin Matuskaworth of buckets have been exhausted.
2663ff01b23SMartin Matuska.
2673ff01b23SMartin Matuska.It Sy metaslab_debug_load Ns = Ns Sy 0 Ns | Ns 1 Pq int
2683ff01b23SMartin MatuskaLoad all metaslabs during pool import.
2693ff01b23SMartin Matuska.
2703ff01b23SMartin Matuska.It Sy metaslab_debug_unload Ns = Ns Sy 0 Ns | Ns 1 Pq int
2713ff01b23SMartin MatuskaPrevent metaslabs from being unloaded.
2723ff01b23SMartin Matuska.
2733ff01b23SMartin Matuska.It Sy metaslab_fragmentation_factor_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
2743ff01b23SMartin MatuskaEnable use of the fragmentation metric in computing metaslab weights.
2753ff01b23SMartin Matuska.
276be181ee2SMartin Matuska.It Sy metaslab_df_max_search Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint
2773ff01b23SMartin MatuskaMaximum distance to search forward from the last offset.
2783ff01b23SMartin MatuskaWithout this limit, fragmented pools can see
2793ff01b23SMartin Matuska.Em >100`000
2803ff01b23SMartin Matuskaiterations and
2813ff01b23SMartin Matuska.Fn metaslab_block_picker
2823ff01b23SMartin Matuskabecomes the performance limiting factor on high-performance storage.
2833ff01b23SMartin Matuska.Pp
2843ff01b23SMartin MatuskaWith the default setting of
285716fd348SMartin Matuska.Sy 16 MiB ,
2863ff01b23SMartin Matuskawe typically see less than
2873ff01b23SMartin Matuska.Em 500
2883ff01b23SMartin Matuskaiterations, even with very fragmented
2893ff01b23SMartin Matuska.Sy ashift Ns = Ns Sy 9
2903ff01b23SMartin Matuskapools.
2913ff01b23SMartin MatuskaThe maximum number of iterations possible is
2923ff01b23SMartin Matuska.Sy metaslab_df_max_search / 2^(ashift+1) .
2933ff01b23SMartin MatuskaWith the default setting of
294716fd348SMartin Matuska.Sy 16 MiB
2953ff01b23SMartin Matuskathis is
2963ff01b23SMartin Matuska.Em 16*1024 Pq with Sy ashift Ns = Ns Sy 9
2973ff01b23SMartin Matuskaor
2983ff01b23SMartin Matuska.Em 2*1024 Pq with Sy ashift Ns = Ns Sy 12 .
2993ff01b23SMartin Matuska.
3003ff01b23SMartin Matuska.It Sy metaslab_df_use_largest_segment Ns = Ns Sy 0 Ns | Ns 1 Pq int
3013ff01b23SMartin MatuskaIf not searching forward (due to
3023ff01b23SMartin Matuska.Sy metaslab_df_max_search , metaslab_df_free_pct ,
3033ff01b23SMartin Matuska.No or Sy metaslab_df_alloc_threshold ) ,
3043ff01b23SMartin Matuskathis tunable controls which segment is used.
3053ff01b23SMartin MatuskaIf set, we will use the largest free segment.
3063ff01b23SMartin MatuskaIf unset, we will use a segment of at least the requested size.
3073ff01b23SMartin Matuska.
308dbd5678dSMartin Matuska.It Sy zfs_metaslab_max_size_cache_sec Ns = Ns Sy 3600 Ns s Po 1 hour Pc Pq u64
3093ff01b23SMartin MatuskaWhen we unload a metaslab, we cache the size of the largest free chunk.
3103ff01b23SMartin MatuskaWe use that cached size to determine whether or not to load a metaslab
3113ff01b23SMartin Matuskafor a given allocation.
3123ff01b23SMartin MatuskaAs more frees accumulate in that metaslab while it's unloaded,
3133ff01b23SMartin Matuskathe cached max size becomes less and less accurate.
3143ff01b23SMartin MatuskaAfter a number of seconds controlled by this tunable,
3153ff01b23SMartin Matuskawe stop considering the cached max size and start
3163ff01b23SMartin Matuskaconsidering only the histogram instead.
3173ff01b23SMartin Matuska.
318be181ee2SMartin Matuska.It Sy zfs_metaslab_mem_limit Ns = Ns Sy 25 Ns % Pq uint
3193ff01b23SMartin MatuskaWhen we are loading a new metaslab, we check the amount of memory being used
3203ff01b23SMartin Matuskato store metaslab range trees.
3213ff01b23SMartin MatuskaIf it is over a threshold, we attempt to unload the least recently used metaslab
3223ff01b23SMartin Matuskato prevent the system from clogging all of its memory with range trees.
3233ff01b23SMartin MatuskaThis tunable sets the percentage of total system memory that is the threshold.
3243ff01b23SMartin Matuska.
3253ff01b23SMartin Matuska.It Sy zfs_metaslab_try_hard_before_gang Ns = Ns Sy 0 Ns | Ns 1 Pq int
3263ff01b23SMartin Matuska.Bl -item -compact
3273ff01b23SMartin Matuska.It
3283ff01b23SMartin MatuskaIf unset, we will first try normal allocation.
3293ff01b23SMartin Matuska.It
3303ff01b23SMartin MatuskaIf that fails then we will do a gang allocation.
3313ff01b23SMartin Matuska.It
3323ff01b23SMartin MatuskaIf that fails then we will do a "try hard" gang allocation.
3333ff01b23SMartin Matuska.It
3343ff01b23SMartin MatuskaIf that fails then we will have a multi-layer gang block.
3353ff01b23SMartin Matuska.El
3363ff01b23SMartin Matuska.Pp
3373ff01b23SMartin Matuska.Bl -item -compact
3383ff01b23SMartin Matuska.It
3393ff01b23SMartin MatuskaIf set, we will first try normal allocation.
3403ff01b23SMartin Matuska.It
3413ff01b23SMartin MatuskaIf that fails then we will do a "try hard" allocation.
3423ff01b23SMartin Matuska.It
3433ff01b23SMartin MatuskaIf that fails we will do a gang allocation.
3443ff01b23SMartin Matuska.It
3453ff01b23SMartin MatuskaIf that fails we will do a "try hard" gang allocation.
3463ff01b23SMartin Matuska.It
3473ff01b23SMartin MatuskaIf that fails then we will have a multi-layer gang block.
3483ff01b23SMartin Matuska.El
3493ff01b23SMartin Matuska.
350be181ee2SMartin Matuska.It Sy zfs_metaslab_find_max_tries Ns = Ns Sy 100 Pq uint
3513ff01b23SMartin MatuskaWhen not trying hard, we only consider this number of the best metaslabs.
3523ff01b23SMartin MatuskaThis improves performance, especially when there are many metaslabs per vdev
3533ff01b23SMartin Matuskaand the allocation can't actually be satisfied
3543ff01b23SMartin Matuska(so we would otherwise iterate all metaslabs).
3553ff01b23SMartin Matuska.
356be181ee2SMartin Matuska.It Sy zfs_vdev_default_ms_count Ns = Ns Sy 200 Pq uint
3573ff01b23SMartin MatuskaWhen a vdev is added, target this number of metaslabs per top-level vdev.
3583ff01b23SMartin Matuska.
359be181ee2SMartin Matuska.It Sy zfs_vdev_default_ms_shift Ns = Ns Sy 29 Po 512 MiB Pc Pq uint
3603ff01b23SMartin MatuskaDefault limit for metaslab size.
3613ff01b23SMartin Matuska.
362dbd5678dSMartin Matuska.It Sy zfs_vdev_max_auto_ashift Ns = Ns Sy 14 Pq uint
363bb2d13b6SMartin MatuskaMaximum ashift used when optimizing for logical \[->] physical sector size on
364bb2d13b6SMartin Matuskanew
3653ff01b23SMartin Matuskatop-level vdevs.
366c7046f76SMartin MatuskaMay be increased up to
367c7046f76SMartin Matuska.Sy ASHIFT_MAX Po 16 Pc ,
368c7046f76SMartin Matuskabut this may negatively impact pool space efficiency.
3693ff01b23SMartin Matuska.
370dbd5678dSMartin Matuska.It Sy zfs_vdev_min_auto_ashift Ns = Ns Sy ASHIFT_MIN Po 9 Pc Pq uint
3713ff01b23SMartin MatuskaMinimum ashift used when creating new top-level vdevs.
3723ff01b23SMartin Matuska.
373be181ee2SMartin Matuska.It Sy zfs_vdev_min_ms_count Ns = Ns Sy 16 Pq uint
3743ff01b23SMartin MatuskaMinimum number of metaslabs to create in a top-level vdev.
3753ff01b23SMartin Matuska.
3763ff01b23SMartin Matuska.It Sy vdev_validate_skip Ns = Ns Sy 0 Ns | Ns 1 Pq int
3773ff01b23SMartin MatuskaSkip label validation steps during pool import.
3783ff01b23SMartin MatuskaChanging is not recommended unless you know what you're doing
3793ff01b23SMartin Matuskaand are recovering a damaged label.
3803ff01b23SMartin Matuska.
381be181ee2SMartin Matuska.It Sy zfs_vdev_ms_count_limit Ns = Ns Sy 131072 Po 128k Pc Pq uint
3823ff01b23SMartin MatuskaPractical upper limit of total metaslabs per top-level vdev.
3833ff01b23SMartin Matuska.
3843ff01b23SMartin Matuska.It Sy metaslab_preload_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
3853ff01b23SMartin MatuskaEnable metaslab group preloading.
3863ff01b23SMartin Matuska.
3873ff01b23SMartin Matuska.It Sy metaslab_lba_weighting_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
3883ff01b23SMartin MatuskaGive more weight to metaslabs with lower LBAs,
3893ff01b23SMartin Matuskaassuming they have greater bandwidth,
3903ff01b23SMartin Matuskaas is typically the case on a modern constant angular velocity disk drive.
3913ff01b23SMartin Matuska.
392be181ee2SMartin Matuska.It Sy metaslab_unload_delay Ns = Ns Sy 32 Pq uint
3933ff01b23SMartin MatuskaAfter a metaslab is used, we keep it loaded for this many TXGs, to attempt to
3943ff01b23SMartin Matuskareduce unnecessary reloading.
3953ff01b23SMartin MatuskaNote that both this many TXGs and
3963ff01b23SMartin Matuska.Sy metaslab_unload_delay_ms
3973ff01b23SMartin Matuskamilliseconds must pass before unloading will occur.
3983ff01b23SMartin Matuska.
399be181ee2SMartin Matuska.It Sy metaslab_unload_delay_ms Ns = Ns Sy 600000 Ns ms Po 10 min Pc Pq uint
4003ff01b23SMartin MatuskaAfter a metaslab is used, we keep it loaded for this many milliseconds,
4013ff01b23SMartin Matuskato attempt to reduce unnecessary reloading.
4023ff01b23SMartin MatuskaNote, that both this many milliseconds and
4033ff01b23SMartin Matuska.Sy metaslab_unload_delay
4043ff01b23SMartin MatuskaTXGs must pass before unloading will occur.
4053ff01b23SMartin Matuska.
406be181ee2SMartin Matuska.It Sy reference_history Ns = Ns Sy 3 Pq uint
407bb2d13b6SMartin MatuskaMaximum reference holders being tracked when reference_tracking_enable is
408bb2d13b6SMartin Matuskaactive.
4093ff01b23SMartin Matuska.
4103ff01b23SMartin Matuska.It Sy reference_tracking_enable Ns = Ns Sy 0 Ns | Ns 1 Pq int
4113ff01b23SMartin MatuskaTrack reference holders to
4123ff01b23SMartin Matuska.Sy refcount_t
4133ff01b23SMartin Matuskaobjects (debug builds only).
4143ff01b23SMartin Matuska.
4153ff01b23SMartin Matuska.It Sy send_holes_without_birth_time Ns = Ns Sy 1 Ns | Ns 0 Pq int
4163ff01b23SMartin MatuskaWhen set, the
4173ff01b23SMartin Matuska.Sy hole_birth
4183ff01b23SMartin Matuskaoptimization will not be used, and all holes will always be sent during a
4193ff01b23SMartin Matuska.Nm zfs Cm send .
4203ff01b23SMartin MatuskaThis is useful if you suspect your datasets are affected by a bug in
4213ff01b23SMartin Matuska.Sy hole_birth .
4223ff01b23SMartin Matuska.
4233ff01b23SMartin Matuska.It Sy spa_config_path Ns = Ns Pa /etc/zfs/zpool.cache Pq charp
4243ff01b23SMartin MatuskaSPA config file.
4253ff01b23SMartin Matuska.
426be181ee2SMartin Matuska.It Sy spa_asize_inflation Ns = Ns Sy 24 Pq uint
4273ff01b23SMartin MatuskaMultiplication factor used to estimate actual disk consumption from the
4283ff01b23SMartin Matuskasize of data being written.
4293ff01b23SMartin MatuskaThe default value is a worst case estimate,
4303ff01b23SMartin Matuskabut lower values may be valid for a given pool depending on its configuration.
4313ff01b23SMartin MatuskaPool administrators who understand the factors involved
4323ff01b23SMartin Matuskamay wish to specify a more realistic inflation factor,
4333ff01b23SMartin Matuskaparticularly if they operate close to quota or capacity limits.
4343ff01b23SMartin Matuska.
4353ff01b23SMartin Matuska.It Sy spa_load_print_vdev_tree Ns = Ns Sy 0 Ns | Ns 1 Pq int
436bb2d13b6SMartin MatuskaWhether to print the vdev tree in the debugging message buffer during pool
437bb2d13b6SMartin Matuskaimport.
4383ff01b23SMartin Matuska.
4393ff01b23SMartin Matuska.It Sy spa_load_verify_data Ns = Ns Sy 1 Ns | Ns 0 Pq int
4403ff01b23SMartin MatuskaWhether to traverse data blocks during an "extreme rewind"
4413ff01b23SMartin Matuska.Pq Fl X
4423ff01b23SMartin Matuskaimport.
4433ff01b23SMartin Matuska.Pp
4443ff01b23SMartin MatuskaAn extreme rewind import normally performs a full traversal of all
4453ff01b23SMartin Matuskablocks in the pool for verification.
4463ff01b23SMartin MatuskaIf this parameter is unset, the traversal skips non-metadata blocks.
4473ff01b23SMartin MatuskaIt can be toggled once the
4483ff01b23SMartin Matuskaimport has started to stop or start the traversal of non-metadata blocks.
4493ff01b23SMartin Matuska.
4503ff01b23SMartin Matuska.It Sy spa_load_verify_metadata  Ns = Ns Sy 1 Ns | Ns 0 Pq int
4513ff01b23SMartin MatuskaWhether to traverse blocks during an "extreme rewind"
4523ff01b23SMartin Matuska.Pq Fl X
4533ff01b23SMartin Matuskapool import.
4543ff01b23SMartin Matuska.Pp
4553ff01b23SMartin MatuskaAn extreme rewind import normally performs a full traversal of all
4563ff01b23SMartin Matuskablocks in the pool for verification.
4573ff01b23SMartin MatuskaIf this parameter is unset, the traversal is not performed.
4583ff01b23SMartin MatuskaIt can be toggled once the import has started to stop or start the traversal.
4593ff01b23SMartin Matuska.
460be181ee2SMartin Matuska.It Sy spa_load_verify_shift Ns = Ns Sy 4 Po 1/16th Pc Pq uint
4613ff01b23SMartin MatuskaSets the maximum number of bytes to consume during pool import to the log2
4623ff01b23SMartin Matuskafraction of the target ARC size.
4633ff01b23SMartin Matuska.
4643ff01b23SMartin Matuska.It Sy spa_slop_shift Ns = Ns Sy 5 Po 1/32nd Pc Pq int
4653ff01b23SMartin MatuskaNormally, we don't allow the last
4663ff01b23SMartin Matuska.Sy 3.2% Pq Sy 1/2^spa_slop_shift
4673ff01b23SMartin Matuskaof space in the pool to be consumed.
4683ff01b23SMartin MatuskaThis ensures that we don't run the pool completely out of space,
4693ff01b23SMartin Matuskadue to unaccounted changes (e.g. to the MOS).
4703ff01b23SMartin MatuskaIt also limits the worst-case time to allocate space.
4713ff01b23SMartin MatuskaIf we have less than this amount of free space,
4723ff01b23SMartin Matuskamost ZPL operations (e.g. write, create) will return
4733ff01b23SMartin Matuska.Sy ENOSPC .
4743ff01b23SMartin Matuska.
475716fd348SMartin Matuska.It Sy spa_upgrade_errlog_limit Ns = Ns Sy 0 Pq uint
476716fd348SMartin MatuskaLimits the number of on-disk error log entries that will be converted to the
477716fd348SMartin Matuskanew format when enabling the
478716fd348SMartin Matuska.Sy head_errlog
479716fd348SMartin Matuskafeature.
480716fd348SMartin MatuskaThe default is to convert all log entries.
481716fd348SMartin Matuska.
482be181ee2SMartin Matuska.It Sy vdev_removal_max_span Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint
4833ff01b23SMartin MatuskaDuring top-level vdev removal, chunks of data are copied from the vdev
4843ff01b23SMartin Matuskawhich may include free space in order to trade bandwidth for IOPS.
4853ff01b23SMartin MatuskaThis parameter determines the maximum span of free space, in bytes,
4863ff01b23SMartin Matuskawhich will be included as "unnecessary" data in a chunk of copied data.
4873ff01b23SMartin Matuska.Pp
4883ff01b23SMartin MatuskaThe default value here was chosen to align with
4893ff01b23SMartin Matuska.Sy zfs_vdev_read_gap_limit ,
4903ff01b23SMartin Matuskawhich is a similar concept when doing
4913ff01b23SMartin Matuskaregular reads (but there's no reason it has to be the same).
4923ff01b23SMartin Matuska.
493dbd5678dSMartin Matuska.It Sy vdev_file_logical_ashift Ns = Ns Sy 9 Po 512 B Pc Pq u64
4943ff01b23SMartin MatuskaLogical ashift for file-based devices.
4953ff01b23SMartin Matuska.
496dbd5678dSMartin Matuska.It Sy vdev_file_physical_ashift Ns = Ns Sy 9 Po 512 B Pc Pq u64
4973ff01b23SMartin MatuskaPhysical ashift for file-based devices.
4983ff01b23SMartin Matuska.
4993ff01b23SMartin Matuska.It Sy zap_iterate_prefetch Ns = Ns Sy 1 Ns | Ns 0 Pq int
5003ff01b23SMartin MatuskaIf set, when we start iterating over a ZAP object,
5013ff01b23SMartin Matuskaprefetch the entire object (all leaf blocks).
5023ff01b23SMartin MatuskaHowever, this is limited by
5033ff01b23SMartin Matuska.Sy dmu_prefetch_max .
5043ff01b23SMartin Matuska.
505*15f0b8c3SMartin Matuska.It Sy zap_micro_max_size Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq int
506*15f0b8c3SMartin MatuskaMaximum micro ZAP size.
507*15f0b8c3SMartin MatuskaA micro ZAP is upgraded to a fat ZAP, once it grows beyond the specified size.
508*15f0b8c3SMartin Matuska.
509dbd5678dSMartin Matuska.It Sy zfetch_array_rd_sz Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64
5103ff01b23SMartin MatuskaIf prefetching is enabled, disable prefetching for reads larger than this size.
5113ff01b23SMartin Matuska.
512e3aa18adSMartin Matuska.It Sy zfetch_min_distance Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq uint
513e3aa18adSMartin MatuskaMin bytes to prefetch per stream.
514e3aa18adSMartin MatuskaPrefetch distance starts from the demand access size and quickly grows to
515e3aa18adSMartin Matuskathis value, doubling on each hit.
516e3aa18adSMartin MatuskaAfter that it may grow further by 1/8 per hit, but only if some prefetch
517e3aa18adSMartin Matuskasince last time haven't completed in time to satisfy demand request, i.e.
518e3aa18adSMartin Matuskaprefetch depth didn't cover the read latency or the pool got saturated.
519e3aa18adSMartin Matuska.
520e3aa18adSMartin Matuska.It Sy zfetch_max_distance Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq uint
5213ff01b23SMartin MatuskaMax bytes to prefetch per stream.
5223ff01b23SMartin Matuska.
523716fd348SMartin Matuska.It Sy zfetch_max_idistance Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq uint
5243ff01b23SMartin MatuskaMax bytes to prefetch indirects for per stream.
5253ff01b23SMartin Matuska.
5263ff01b23SMartin Matuska.It Sy zfetch_max_streams Ns = Ns Sy 8 Pq uint
5273ff01b23SMartin MatuskaMax number of streams per zfetch (prefetch streams per file).
5283ff01b23SMartin Matuska.
529e3aa18adSMartin Matuska.It Sy zfetch_min_sec_reap Ns = Ns Sy 1 Pq uint
530e3aa18adSMartin MatuskaMin time before inactive prefetch stream can be reclaimed
531e3aa18adSMartin Matuska.
532e3aa18adSMartin Matuska.It Sy zfetch_max_sec_reap Ns = Ns Sy 2 Pq uint
533e3aa18adSMartin MatuskaMax time before inactive prefetch stream can be deleted
5343ff01b23SMartin Matuska.
5353ff01b23SMartin Matuska.It Sy zfs_abd_scatter_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
5363ff01b23SMartin MatuskaEnables ARC from using scatter/gather lists and forces all allocations to be
5373ff01b23SMartin Matuskalinear in kernel memory.
5383ff01b23SMartin MatuskaDisabling can improve performance in some code paths
5393ff01b23SMartin Matuskaat the expense of fragmented kernel memory.
5403ff01b23SMartin Matuska.
541e92ffd9bSMartin Matuska.It Sy zfs_abd_scatter_max_order Ns = Ns Sy MAX_ORDER\-1 Pq uint
5423ff01b23SMartin MatuskaMaximum number of consecutive memory pages allocated in a single block for
5433ff01b23SMartin Matuskascatter/gather lists.
5443ff01b23SMartin Matuska.Pp
5453ff01b23SMartin MatuskaThe value of
5463ff01b23SMartin Matuska.Sy MAX_ORDER
5473ff01b23SMartin Matuskadepends on kernel configuration.
5483ff01b23SMartin Matuska.
549716fd348SMartin Matuska.It Sy zfs_abd_scatter_min_size Ns = Ns Sy 1536 Ns B Po 1.5 KiB Pc Pq uint
5503ff01b23SMartin MatuskaThis is the minimum allocation size that will use scatter (page-based) ABDs.
5513ff01b23SMartin MatuskaSmaller allocations will use linear ABDs.
5523ff01b23SMartin Matuska.
553dbd5678dSMartin Matuska.It Sy zfs_arc_dnode_limit Ns = Ns Sy 0 Ns B Pq u64
5543ff01b23SMartin MatuskaWhen the number of bytes consumed by dnodes in the ARC exceeds this number of
5553ff01b23SMartin Matuskabytes, try to unpin some of it in response to demand for non-metadata.
5563ff01b23SMartin MatuskaThis value acts as a ceiling to the amount of dnode metadata, and defaults to
5573ff01b23SMartin Matuska.Sy 0 ,
5583ff01b23SMartin Matuskawhich indicates that a percent which is based on
5593ff01b23SMartin Matuska.Sy zfs_arc_dnode_limit_percent
5603ff01b23SMartin Matuskaof the ARC meta buffers that may be used for dnodes.
5613ff01b23SMartin Matuska.Pp
5623ff01b23SMartin MatuskaAlso see
5633ff01b23SMartin Matuska.Sy zfs_arc_meta_prune
5643ff01b23SMartin Matuskawhich serves a similar purpose but is used
5653ff01b23SMartin Matuskawhen the amount of metadata in the ARC exceeds
5663ff01b23SMartin Matuska.Sy zfs_arc_meta_limit
5673ff01b23SMartin Matuskarather than in response to overall demand for non-metadata.
5683ff01b23SMartin Matuska.
569dbd5678dSMartin Matuska.It Sy zfs_arc_dnode_limit_percent Ns = Ns Sy 10 Ns % Pq u64
5703ff01b23SMartin MatuskaPercentage that can be consumed by dnodes of ARC meta buffers.
5713ff01b23SMartin Matuska.Pp
5723ff01b23SMartin MatuskaSee also
5733ff01b23SMartin Matuska.Sy zfs_arc_dnode_limit ,
5743ff01b23SMartin Matuskawhich serves a similar purpose but has a higher priority if nonzero.
5753ff01b23SMartin Matuska.
576dbd5678dSMartin Matuska.It Sy zfs_arc_dnode_reduce_percent Ns = Ns Sy 10 Ns % Pq u64
5773ff01b23SMartin MatuskaPercentage of ARC dnodes to try to scan in response to demand for non-metadata
5783ff01b23SMartin Matuskawhen the number of bytes consumed by dnodes exceeds
5793ff01b23SMartin Matuska.Sy zfs_arc_dnode_limit .
5803ff01b23SMartin Matuska.
581be181ee2SMartin Matuska.It Sy zfs_arc_average_blocksize Ns = Ns Sy 8192 Ns B Po 8 KiB Pc Pq uint
5823ff01b23SMartin MatuskaThe ARC's buffer hash table is sized based on the assumption of an average
5833ff01b23SMartin Matuskablock size of this value.
584716fd348SMartin MatuskaThis works out to roughly 1 MiB of hash table per 1 GiB of physical memory
5853ff01b23SMartin Matuskawith 8-byte pointers.
5863ff01b23SMartin MatuskaFor configurations with a known larger average block size,
5873ff01b23SMartin Matuskathis value can be increased to reduce the memory footprint.
5883ff01b23SMartin Matuska.
589be181ee2SMartin Matuska.It Sy zfs_arc_eviction_pct Ns = Ns Sy 200 Ns % Pq uint
5903ff01b23SMartin MatuskaWhen
5913ff01b23SMartin Matuska.Fn arc_is_overflowing ,
5923ff01b23SMartin Matuska.Fn arc_get_data_impl
5933ff01b23SMartin Matuskawaits for this percent of the requested amount of data to be evicted.
5943ff01b23SMartin MatuskaFor example, by default, for every
595716fd348SMartin Matuska.Em 2 KiB
5963ff01b23SMartin Matuskathat's evicted,
597716fd348SMartin Matuska.Em 1 KiB
5983ff01b23SMartin Matuskaof it may be "reused" by a new allocation.
5993ff01b23SMartin MatuskaSince this is above
6003ff01b23SMartin Matuska.Sy 100 Ns % ,
6013ff01b23SMartin Matuskait ensures that progress is made towards getting
6023ff01b23SMartin Matuska.Sy arc_size No under Sy arc_c .
6033ff01b23SMartin MatuskaSince this is finite, it ensures that allocations can still happen,
6043ff01b23SMartin Matuskaeven during the potentially long time that
6053ff01b23SMartin Matuska.Sy arc_size No is more than Sy arc_c .
6063ff01b23SMartin Matuska.
607be181ee2SMartin Matuska.It Sy zfs_arc_evict_batch_limit Ns = Ns Sy 10 Pq uint
6083ff01b23SMartin MatuskaNumber ARC headers to evict per sub-list before proceeding to another sub-list.
6093ff01b23SMartin MatuskaThis batch-style operation prevents entire sub-lists from being evicted at once
6103ff01b23SMartin Matuskabut comes at a cost of additional unlocking and locking.
6113ff01b23SMartin Matuska.
612be181ee2SMartin Matuska.It Sy zfs_arc_grow_retry Ns = Ns Sy 0 Ns s Pq uint
6133ff01b23SMartin MatuskaIf set to a non zero value, it will replace the
6143ff01b23SMartin Matuska.Sy arc_grow_retry
6153ff01b23SMartin Matuskavalue with this value.
6163ff01b23SMartin MatuskaThe
6173ff01b23SMartin Matuska.Sy arc_grow_retry
6183ff01b23SMartin Matuska.No value Pq default Sy 5 Ns s
6193ff01b23SMartin Matuskais the number of seconds the ARC will wait before
6203ff01b23SMartin Matuskatrying to resume growth after a memory pressure event.
6213ff01b23SMartin Matuska.
6223ff01b23SMartin Matuska.It Sy zfs_arc_lotsfree_percent Ns = Ns Sy 10 Ns % Pq int
6233ff01b23SMartin MatuskaThrottle I/O when free system memory drops below this percentage of total
6243ff01b23SMartin Matuskasystem memory.
6253ff01b23SMartin MatuskaSetting this value to
6263ff01b23SMartin Matuska.Sy 0
6273ff01b23SMartin Matuskawill disable the throttle.
6283ff01b23SMartin Matuska.
629dbd5678dSMartin Matuska.It Sy zfs_arc_max Ns = Ns Sy 0 Ns B Pq u64
6303ff01b23SMartin MatuskaMax size of ARC in bytes.
6313ff01b23SMartin MatuskaIf
6323ff01b23SMartin Matuska.Sy 0 ,
6333ff01b23SMartin Matuskathen the max size of ARC is determined by the amount of system memory installed.
6343ff01b23SMartin MatuskaUnder Linux, half of system memory will be used as the limit.
6353ff01b23SMartin MatuskaUnder
6363ff01b23SMartin Matuska.Fx ,
6373ff01b23SMartin Matuskathe larger of
638716fd348SMartin Matuska.Sy all_system_memory No \- Sy 1 GiB
639e92ffd9bSMartin Matuskaand
640e92ffd9bSMartin Matuska.Sy 5/8 No \(mu Sy all_system_memory
6413ff01b23SMartin Matuskawill be used as the limit.
6423ff01b23SMartin MatuskaThis value must be at least
643716fd348SMartin Matuska.Sy 67108864 Ns B Pq 64 MiB .
6443ff01b23SMartin Matuska.Pp
6453ff01b23SMartin MatuskaThis value can be changed dynamically, with some caveats.
6463ff01b23SMartin MatuskaIt cannot be set back to
6473ff01b23SMartin Matuska.Sy 0
6483ff01b23SMartin Matuskawhile running, and reducing it below the current ARC size will not cause
6493ff01b23SMartin Matuskathe ARC to shrink without memory pressure to induce shrinking.
6503ff01b23SMartin Matuska.
651be181ee2SMartin Matuska.It Sy zfs_arc_meta_adjust_restarts Ns = Ns Sy 4096 Pq uint
6523ff01b23SMartin MatuskaThe number of restart passes to make while scanning the ARC attempting
6533ff01b23SMartin Matuskathe free buffers in order to stay below the
6543ff01b23SMartin Matuska.Sy fs_arc_meta_limit .
6553ff01b23SMartin MatuskaThis value should not need to be tuned but is available to facilitate
6563ff01b23SMartin Matuskaperformance analysis.
6573ff01b23SMartin Matuska.
658dbd5678dSMartin Matuska.It Sy zfs_arc_meta_limit Ns = Ns Sy 0 Ns B Pq u64
6593ff01b23SMartin MatuskaThe maximum allowed size in bytes that metadata buffers are allowed to
6603ff01b23SMartin Matuskaconsume in the ARC.
6613ff01b23SMartin MatuskaWhen this limit is reached, metadata buffers will be reclaimed,
6623ff01b23SMartin Matuskaeven if the overall
6633ff01b23SMartin Matuska.Sy arc_c_max
6643ff01b23SMartin Matuskahas not been reached.
6653ff01b23SMartin MatuskaIt defaults to
6663ff01b23SMartin Matuska.Sy 0 ,
6673ff01b23SMartin Matuskawhich indicates that a percentage based on
6683ff01b23SMartin Matuska.Sy zfs_arc_meta_limit_percent
6693ff01b23SMartin Matuskaof the ARC may be used for metadata.
6703ff01b23SMartin Matuska.Pp
671bb2d13b6SMartin MatuskaThis value my be changed dynamically, except that must be set to an explicit
672bb2d13b6SMartin Matuskavalue
6733ff01b23SMartin Matuska.Pq cannot be set back to Sy 0 .
6743ff01b23SMartin Matuska.
675dbd5678dSMartin Matuska.It Sy zfs_arc_meta_limit_percent Ns = Ns Sy 75 Ns % Pq u64
6763ff01b23SMartin MatuskaPercentage of ARC buffers that can be used for metadata.
6773ff01b23SMartin Matuska.Pp
6783ff01b23SMartin MatuskaSee also
6793ff01b23SMartin Matuska.Sy zfs_arc_meta_limit ,
6803ff01b23SMartin Matuskawhich serves a similar purpose but has a higher priority if nonzero.
6813ff01b23SMartin Matuska.
682dbd5678dSMartin Matuska.It Sy zfs_arc_meta_min Ns = Ns Sy 0 Ns B Pq u64
6833ff01b23SMartin MatuskaThe minimum allowed size in bytes that metadata buffers may consume in
6843ff01b23SMartin Matuskathe ARC.
6853ff01b23SMartin Matuska.
6863ff01b23SMartin Matuska.It Sy zfs_arc_meta_prune Ns = Ns Sy 10000 Pq int
6873ff01b23SMartin MatuskaThe number of dentries and inodes to be scanned looking for entries
6883ff01b23SMartin Matuskawhich can be dropped.
6893ff01b23SMartin MatuskaThis may be required when the ARC reaches the
6903ff01b23SMartin Matuska.Sy zfs_arc_meta_limit
6913ff01b23SMartin Matuskabecause dentries and inodes can pin buffers in the ARC.
6923ff01b23SMartin MatuskaIncreasing this value will cause to dentry and inode caches
6933ff01b23SMartin Matuskato be pruned more aggressively.
6943ff01b23SMartin MatuskaSetting this value to
6953ff01b23SMartin Matuska.Sy 0
6963ff01b23SMartin Matuskawill disable pruning the inode and dentry caches.
6973ff01b23SMartin Matuska.
698be181ee2SMartin Matuska.It Sy zfs_arc_meta_strategy Ns = Ns Sy 1 Ns | Ns 0 Pq uint
6993ff01b23SMartin MatuskaDefine the strategy for ARC metadata buffer eviction (meta reclaim strategy):
7003ff01b23SMartin Matuska.Bl -tag -compact -offset 4n -width "0 (META_ONLY)"
7013ff01b23SMartin Matuska.It Sy 0 Pq META_ONLY
7023ff01b23SMartin Matuskaevict only the ARC metadata buffers
7033ff01b23SMartin Matuska.It Sy 1 Pq BALANCED
7043ff01b23SMartin Matuskaadditional data buffers may be evicted if required
7053ff01b23SMartin Matuskato evict the required number of metadata buffers.
7063ff01b23SMartin Matuska.El
7073ff01b23SMartin Matuska.
708dbd5678dSMartin Matuska.It Sy zfs_arc_min Ns = Ns Sy 0 Ns B Pq u64
7093ff01b23SMartin MatuskaMin size of ARC in bytes.
7103ff01b23SMartin Matuska.No If set to Sy 0 , arc_c_min
7113ff01b23SMartin Matuskawill default to consuming the larger of
712716fd348SMartin Matuska.Sy 32 MiB
713e92ffd9bSMartin Matuskaand
714e92ffd9bSMartin Matuska.Sy all_system_memory No / Sy 32 .
7153ff01b23SMartin Matuska.
716be181ee2SMartin Matuska.It Sy zfs_arc_min_prefetch_ms Ns = Ns Sy 0 Ns ms Ns Po Ns ≡ Ns 1s Pc Pq uint
7173ff01b23SMartin MatuskaMinimum time prefetched blocks are locked in the ARC.
7183ff01b23SMartin Matuska.
719be181ee2SMartin Matuska.It Sy zfs_arc_min_prescient_prefetch_ms Ns = Ns Sy 0 Ns ms Ns Po Ns ≡ Ns 6s Pc Pq uint
7203ff01b23SMartin MatuskaMinimum time "prescient prefetched" blocks are locked in the ARC.
7213ff01b23SMartin MatuskaThese blocks are meant to be prefetched fairly aggressively ahead of
7223ff01b23SMartin Matuskathe code that may use them.
7233ff01b23SMartin Matuska.
724e92ffd9bSMartin Matuska.It Sy zfs_arc_prune_task_threads Ns = Ns Sy 1 Pq int
725e92ffd9bSMartin MatuskaNumber of arc_prune threads.
726e92ffd9bSMartin Matuska.Fx
727e92ffd9bSMartin Matuskadoes not need more than one.
728e92ffd9bSMartin MatuskaLinux may theoretically use one per mount point up to number of CPUs,
729e92ffd9bSMartin Matuskabut that was not proven to be useful.
730e92ffd9bSMartin Matuska.
7313ff01b23SMartin Matuska.It Sy zfs_max_missing_tvds Ns = Ns Sy 0 Pq int
7323ff01b23SMartin MatuskaNumber of missing top-level vdevs which will be allowed during
7333ff01b23SMartin Matuskapool import (only in read-only mode).
7343ff01b23SMartin Matuska.
735dbd5678dSMartin Matuska.It Sy zfs_max_nvlist_src_size Ns = Sy 0 Pq u64
7363ff01b23SMartin MatuskaMaximum size in bytes allowed to be passed as
7373ff01b23SMartin Matuska.Sy zc_nvlist_src_size
7383ff01b23SMartin Matuskafor ioctls on
7393ff01b23SMartin Matuska.Pa /dev/zfs .
7403ff01b23SMartin MatuskaThis prevents a user from causing the kernel to allocate
7413ff01b23SMartin Matuskaan excessive amount of memory.
7423ff01b23SMartin MatuskaWhen the limit is exceeded, the ioctl fails with
7433ff01b23SMartin Matuska.Sy EINVAL
7443ff01b23SMartin Matuskaand a description of the error is sent to the
7453ff01b23SMartin Matuska.Pa zfs-dbgmsg
7463ff01b23SMartin Matuskalog.
7473ff01b23SMartin MatuskaThis parameter should not need to be touched under normal circumstances.
7483ff01b23SMartin MatuskaIf
7493ff01b23SMartin Matuska.Sy 0 ,
7503ff01b23SMartin Matuskaequivalent to a quarter of the user-wired memory limit under
7513ff01b23SMartin Matuska.Fx
7523ff01b23SMartin Matuskaand to
753716fd348SMartin Matuska.Sy 134217728 Ns B Pq 128 MiB
7543ff01b23SMartin Matuskaunder Linux.
7553ff01b23SMartin Matuska.
756be181ee2SMartin Matuska.It Sy zfs_multilist_num_sublists Ns = Ns Sy 0 Pq uint
7573ff01b23SMartin MatuskaTo allow more fine-grained locking, each ARC state contains a series
7583ff01b23SMartin Matuskaof lists for both data and metadata objects.
7593ff01b23SMartin MatuskaLocking is performed at the level of these "sub-lists".
7603ff01b23SMartin MatuskaThis parameters controls the number of sub-lists per ARC state,
7613ff01b23SMartin Matuskaand also applies to other uses of the multilist data structure.
7623ff01b23SMartin Matuska.Pp
7633ff01b23SMartin MatuskaIf
7643ff01b23SMartin Matuska.Sy 0 ,
7653ff01b23SMartin Matuskaequivalent to the greater of the number of online CPUs and
7663ff01b23SMartin Matuska.Sy 4 .
7673ff01b23SMartin Matuska.
7683ff01b23SMartin Matuska.It Sy zfs_arc_overflow_shift Ns = Ns Sy 8 Pq int
7693ff01b23SMartin MatuskaThe ARC size is considered to be overflowing if it exceeds the current
7703ff01b23SMartin MatuskaARC target size
7713ff01b23SMartin Matuska.Pq Sy arc_c
7723f9d360cSMartin Matuskaby thresholds determined by this parameter.
7733f9d360cSMartin MatuskaExceeding by
774e92ffd9bSMartin Matuska.Sy ( arc_c No >> Sy zfs_arc_overflow_shift ) No / Sy 2
7753f9d360cSMartin Matuskastarts ARC reclamation process.
7763f9d360cSMartin MatuskaIf that appears insufficient, exceeding by
777e92ffd9bSMartin Matuska.Sy ( arc_c No >> Sy zfs_arc_overflow_shift ) No \(mu Sy 1.5
7783f9d360cSMartin Matuskablocks new buffer allocation until the reclaim thread catches up.
7793f9d360cSMartin MatuskaStarted reclamation process continues till ARC size returns below the
7803f9d360cSMartin Matuskatarget size.
7813ff01b23SMartin Matuska.Pp
7823ff01b23SMartin MatuskaThe default value of
7833ff01b23SMartin Matuska.Sy 8
7843f9d360cSMartin Matuskacauses the ARC to start reclamation if it exceeds the target size by
7853f9d360cSMartin Matuska.Em 0.2%
7863f9d360cSMartin Matuskaof the target size, and block allocations by
7873f9d360cSMartin Matuska.Em 0.6% .
7883ff01b23SMartin Matuska.
789be181ee2SMartin Matuska.It Sy zfs_arc_p_min_shift Ns = Ns Sy 0 Pq uint
7903ff01b23SMartin MatuskaIf nonzero, this will update
7913ff01b23SMartin Matuska.Sy arc_p_min_shift Pq default Sy 4
7923ff01b23SMartin Matuskawith the new value.
7933ff01b23SMartin Matuska.Sy arc_p_min_shift No is used as a shift of Sy arc_c
7943ff01b23SMartin Matuskawhen calculating the minumum
7953ff01b23SMartin Matuska.Sy arc_p No size .
7963ff01b23SMartin Matuska.
7973ff01b23SMartin Matuska.It Sy zfs_arc_p_dampener_disable Ns = Ns Sy 1 Ns | Ns 0 Pq int
7983ff01b23SMartin MatuskaDisable
7993ff01b23SMartin Matuska.Sy arc_p
8003ff01b23SMartin Matuskaadapt dampener, which reduces the maximum single adjustment to
8013ff01b23SMartin Matuska.Sy arc_p .
8023ff01b23SMartin Matuska.
803be181ee2SMartin Matuska.It Sy zfs_arc_shrink_shift Ns = Ns Sy 0 Pq uint
8043ff01b23SMartin MatuskaIf nonzero, this will update
8053ff01b23SMartin Matuska.Sy arc_shrink_shift Pq default Sy 7
8063ff01b23SMartin Matuskawith the new value.
8073ff01b23SMartin Matuska.
8083ff01b23SMartin Matuska.It Sy zfs_arc_pc_percent Ns = Ns Sy 0 Ns % Po off Pc Pq uint
8093ff01b23SMartin MatuskaPercent of pagecache to reclaim ARC to.
8103ff01b23SMartin Matuska.Pp
8113ff01b23SMartin MatuskaThis tunable allows the ZFS ARC to play more nicely
8123ff01b23SMartin Matuskawith the kernel's LRU pagecache.
8133ff01b23SMartin MatuskaIt can guarantee that the ARC size won't collapse under scanning
8143ff01b23SMartin Matuskapressure on the pagecache, yet still allows the ARC to be reclaimed down to
8153ff01b23SMartin Matuska.Sy zfs_arc_min
8163ff01b23SMartin Matuskaif necessary.
8173ff01b23SMartin MatuskaThis value is specified as percent of pagecache size (as measured by
8183ff01b23SMartin Matuska.Sy NR_FILE_PAGES ) ,
8193ff01b23SMartin Matuskawhere that percent may exceed
8203ff01b23SMartin Matuska.Sy 100 .
8213ff01b23SMartin MatuskaThis
8223ff01b23SMartin Matuskaonly operates during memory pressure/reclaim.
8233ff01b23SMartin Matuska.
8243ff01b23SMartin Matuska.It Sy zfs_arc_shrinker_limit Ns = Ns Sy 10000 Pq int
8253ff01b23SMartin MatuskaThis is a limit on how many pages the ARC shrinker makes available for
8263ff01b23SMartin Matuskaeviction in response to one page allocation attempt.
8273ff01b23SMartin MatuskaNote that in practice, the kernel's shrinker can ask us to evict
8283ff01b23SMartin Matuskaup to about four times this for one allocation attempt.
8293ff01b23SMartin Matuska.Pp
8303ff01b23SMartin MatuskaThe default limit of
831716fd348SMartin Matuska.Sy 10000 Pq in practice, Em 160 MiB No per allocation attempt with 4 KiB pages
8323ff01b23SMartin Matuskalimits the amount of time spent attempting to reclaim ARC memory to
8333ff01b23SMartin Matuskaless than 100 ms per allocation attempt,
834716fd348SMartin Matuskaeven with a small average compressed block size of ~8 KiB.
8353ff01b23SMartin Matuska.Pp
8363ff01b23SMartin MatuskaThe parameter can be set to 0 (zero) to disable the limit,
8373ff01b23SMartin Matuskaand only applies on Linux.
8383ff01b23SMartin Matuska.
839dbd5678dSMartin Matuska.It Sy zfs_arc_sys_free Ns = Ns Sy 0 Ns B Pq u64
8403ff01b23SMartin MatuskaThe target number of bytes the ARC should leave as free memory on the system.
8413ff01b23SMartin MatuskaIf zero, equivalent to the bigger of
842716fd348SMartin Matuska.Sy 512 KiB No and Sy all_system_memory/64 .
8433ff01b23SMartin Matuska.
8443ff01b23SMartin Matuska.It Sy zfs_autoimport_disable Ns = Ns Sy 1 Ns | Ns 0 Pq int
8453ff01b23SMartin MatuskaDisable pool import at module load by ignoring the cache file
8463ff01b23SMartin Matuska.Pq Sy spa_config_path .
8473ff01b23SMartin Matuska.
8483ff01b23SMartin Matuska.It Sy zfs_checksum_events_per_second Ns = Ns Sy 20 Ns /s Pq uint
8493ff01b23SMartin MatuskaRate limit checksum events to this many per second.
8503ff01b23SMartin MatuskaNote that this should not be set below the ZED thresholds
8513ff01b23SMartin Matuska(currently 10 checksums over 10 seconds)
8523ff01b23SMartin Matuskaor else the daemon may not trigger any action.
8533ff01b23SMartin Matuska.
854be181ee2SMartin Matuska.It Sy zfs_commit_timeout_pct Ns = Ns Sy 5 Ns % Pq uint
8553ff01b23SMartin MatuskaThis controls the amount of time that a ZIL block (lwb) will remain "open"
8563ff01b23SMartin Matuskawhen it isn't "full", and it has a thread waiting for it to be committed to
8573ff01b23SMartin Matuskastable storage.
8583ff01b23SMartin MatuskaThe timeout is scaled based on a percentage of the last lwb
8593ff01b23SMartin Matuskalatency to avoid significantly impacting the latency of each individual
8603ff01b23SMartin Matuskatransaction record (itx).
8613ff01b23SMartin Matuska.
8623ff01b23SMartin Matuska.It Sy zfs_condense_indirect_commit_entry_delay_ms Ns = Ns Sy 0 Ns ms Pq int
8633ff01b23SMartin MatuskaVdev indirection layer (used for device removal) sleeps for this many
8643ff01b23SMartin Matuskamilliseconds during mapping generation.
8653ff01b23SMartin MatuskaIntended for use with the test suite to throttle vdev removal speed.
8663ff01b23SMartin Matuska.
867be181ee2SMartin Matuska.It Sy zfs_condense_indirect_obsolete_pct Ns = Ns Sy 25 Ns % Pq uint
868bb2d13b6SMartin MatuskaMinimum percent of obsolete bytes in vdev mapping required to attempt to
869bb2d13b6SMartin Matuskacondense
8703ff01b23SMartin Matuska.Pq see Sy zfs_condense_indirect_vdevs_enable .
8713ff01b23SMartin MatuskaIntended for use with the test suite
8723ff01b23SMartin Matuskato facilitate triggering condensing as needed.
8733ff01b23SMartin Matuska.
8743ff01b23SMartin Matuska.It Sy zfs_condense_indirect_vdevs_enable Ns = Ns Sy 1 Ns | Ns 0 Pq int
8753ff01b23SMartin MatuskaEnable condensing indirect vdev mappings.
8763ff01b23SMartin MatuskaWhen set, attempt to condense indirect vdev mappings
8773ff01b23SMartin Matuskaif the mapping uses more than
8783ff01b23SMartin Matuska.Sy zfs_condense_min_mapping_bytes
8793ff01b23SMartin Matuskabytes of memory and if the obsolete space map object uses more than
8803ff01b23SMartin Matuska.Sy zfs_condense_max_obsolete_bytes
8813ff01b23SMartin Matuskabytes on-disk.
882bb2d13b6SMartin MatuskaThe condensing process is an attempt to save memory by removing obsolete
883bb2d13b6SMartin Matuskamappings.
8843ff01b23SMartin Matuska.
885dbd5678dSMartin Matuska.It Sy zfs_condense_max_obsolete_bytes Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq u64
8863ff01b23SMartin MatuskaOnly attempt to condense indirect vdev mappings if the on-disk size
8873ff01b23SMartin Matuskaof the obsolete space map object is greater than this number of bytes
8883ff01b23SMartin Matuska.Pq see Sy zfs_condense_indirect_vdevs_enable .
8893ff01b23SMartin Matuska.
890dbd5678dSMartin Matuska.It Sy zfs_condense_min_mapping_bytes Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq u64
8913ff01b23SMartin MatuskaMinimum size vdev mapping to attempt to condense
8923ff01b23SMartin Matuska.Pq see Sy zfs_condense_indirect_vdevs_enable .
8933ff01b23SMartin Matuska.
8943ff01b23SMartin Matuska.It Sy zfs_dbgmsg_enable Ns = Ns Sy 1 Ns | Ns 0 Pq int
8953ff01b23SMartin MatuskaInternally ZFS keeps a small log to facilitate debugging.
8963ff01b23SMartin MatuskaThe log is enabled by default, and can be disabled by unsetting this option.
8973ff01b23SMartin MatuskaThe contents of the log can be accessed by reading
8983ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs/dbgmsg .
8993ff01b23SMartin MatuskaWriting
9003ff01b23SMartin Matuska.Sy 0
9013ff01b23SMartin Matuskato the file clears the log.
9023ff01b23SMartin Matuska.Pp
9033ff01b23SMartin MatuskaThis setting does not influence debug prints due to
9043ff01b23SMartin Matuska.Sy zfs_flags .
9053ff01b23SMartin Matuska.
906be181ee2SMartin Matuska.It Sy zfs_dbgmsg_maxsize Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq uint
9073ff01b23SMartin MatuskaMaximum size of the internal ZFS debug log.
9083ff01b23SMartin Matuska.
9093ff01b23SMartin Matuska.It Sy zfs_dbuf_state_index Ns = Ns Sy 0 Pq int
9103ff01b23SMartin MatuskaHistorically used for controlling what reporting was available under
9113ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs .
9123ff01b23SMartin MatuskaNo effect.
9133ff01b23SMartin Matuska.
9143ff01b23SMartin Matuska.It Sy zfs_deadman_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
9153ff01b23SMartin MatuskaWhen a pool sync operation takes longer than
9163ff01b23SMartin Matuska.Sy zfs_deadman_synctime_ms ,
9173ff01b23SMartin Matuskaor when an individual I/O operation takes longer than
9183ff01b23SMartin Matuska.Sy zfs_deadman_ziotime_ms ,
9193ff01b23SMartin Matuskathen the operation is considered to be "hung".
9203ff01b23SMartin MatuskaIf
9213ff01b23SMartin Matuska.Sy zfs_deadman_enabled
9223ff01b23SMartin Matuskais set, then the deadman behavior is invoked as described by
9233ff01b23SMartin Matuska.Sy zfs_deadman_failmode .
9243ff01b23SMartin MatuskaBy default, the deadman is enabled and set to
9253ff01b23SMartin Matuska.Sy wait
926c03c5b1cSMartin Matuskawhich results in "hung" I/O operations only being logged.
9273ff01b23SMartin MatuskaThe deadman is automatically disabled when a pool gets suspended.
9283ff01b23SMartin Matuska.
9293ff01b23SMartin Matuska.It Sy zfs_deadman_failmode Ns = Ns Sy wait Pq charp
9303ff01b23SMartin MatuskaControls the failure behavior when the deadman detects a "hung" I/O operation.
9313ff01b23SMartin MatuskaValid values are:
9323ff01b23SMartin Matuska.Bl -tag -compact -offset 4n -width "continue"
9333ff01b23SMartin Matuska.It Sy wait
9343ff01b23SMartin MatuskaWait for a "hung" operation to complete.
9353ff01b23SMartin MatuskaFor each "hung" operation a "deadman" event will be posted
9363ff01b23SMartin Matuskadescribing that operation.
9373ff01b23SMartin Matuska.It Sy continue
9383ff01b23SMartin MatuskaAttempt to recover from a "hung" operation by re-dispatching it
9393ff01b23SMartin Matuskato the I/O pipeline if possible.
9403ff01b23SMartin Matuska.It Sy panic
9413ff01b23SMartin MatuskaPanic the system.
9423ff01b23SMartin MatuskaThis can be used to facilitate automatic fail-over
9433ff01b23SMartin Matuskato a properly configured fail-over partner.
9443ff01b23SMartin Matuska.El
9453ff01b23SMartin Matuska.
946dbd5678dSMartin Matuska.It Sy zfs_deadman_checktime_ms Ns = Ns Sy 60000 Ns ms Po 1 min Pc Pq u64
9473ff01b23SMartin MatuskaCheck time in milliseconds.
9483ff01b23SMartin MatuskaThis defines the frequency at which we check for hung I/O requests
9493ff01b23SMartin Matuskaand potentially invoke the
9503ff01b23SMartin Matuska.Sy zfs_deadman_failmode
9513ff01b23SMartin Matuskabehavior.
9523ff01b23SMartin Matuska.
953dbd5678dSMartin Matuska.It Sy zfs_deadman_synctime_ms Ns = Ns Sy 600000 Ns ms Po 10 min Pc Pq u64
9543ff01b23SMartin MatuskaInterval in milliseconds after which the deadman is triggered and also
9553ff01b23SMartin Matuskathe interval after which a pool sync operation is considered to be "hung".
9563ff01b23SMartin MatuskaOnce this limit is exceeded the deadman will be invoked every
9573ff01b23SMartin Matuska.Sy zfs_deadman_checktime_ms
9583ff01b23SMartin Matuskamilliseconds until the pool sync completes.
9593ff01b23SMartin Matuska.
960dbd5678dSMartin Matuska.It Sy zfs_deadman_ziotime_ms Ns = Ns Sy 300000 Ns ms Po 5 min Pc Pq u64
9613ff01b23SMartin MatuskaInterval in milliseconds after which the deadman is triggered and an
9623ff01b23SMartin Matuskaindividual I/O operation is considered to be "hung".
9633ff01b23SMartin MatuskaAs long as the operation remains "hung",
9643ff01b23SMartin Matuskathe deadman will be invoked every
9653ff01b23SMartin Matuska.Sy zfs_deadman_checktime_ms
9663ff01b23SMartin Matuskamilliseconds until the operation completes.
9673ff01b23SMartin Matuska.
9683ff01b23SMartin Matuska.It Sy zfs_dedup_prefetch Ns = Ns Sy 0 Ns | Ns 1 Pq int
9693ff01b23SMartin MatuskaEnable prefetching dedup-ed blocks which are going to be freed.
9703ff01b23SMartin Matuska.
971be181ee2SMartin Matuska.It Sy zfs_delay_min_dirty_percent Ns = Ns Sy 60 Ns % Pq uint
9723ff01b23SMartin MatuskaStart to delay each transaction once there is this amount of dirty data,
9733ff01b23SMartin Matuskaexpressed as a percentage of
9743ff01b23SMartin Matuska.Sy zfs_dirty_data_max .
9753ff01b23SMartin MatuskaThis value should be at least
9763ff01b23SMartin Matuska.Sy zfs_vdev_async_write_active_max_dirty_percent .
9773ff01b23SMartin Matuska.No See Sx ZFS TRANSACTION DELAY .
9783ff01b23SMartin Matuska.
9793ff01b23SMartin Matuska.It Sy zfs_delay_scale Ns = Ns Sy 500000 Pq int
9803ff01b23SMartin MatuskaThis controls how quickly the transaction delay approaches infinity.
9813ff01b23SMartin MatuskaLarger values cause longer delays for a given amount of dirty data.
9823ff01b23SMartin Matuska.Pp
9833ff01b23SMartin MatuskaFor the smoothest delay, this value should be about 1 billion divided
9843ff01b23SMartin Matuskaby the maximum number of operations per second.
9853ff01b23SMartin MatuskaThis will smoothly handle between ten times and a tenth of this number.
9863ff01b23SMartin Matuska.No See Sx ZFS TRANSACTION DELAY .
9873ff01b23SMartin Matuska.Pp
988e92ffd9bSMartin Matuska.Sy zfs_delay_scale No \(mu Sy zfs_dirty_data_max Em must No be smaller than Sy 2^64 .
9893ff01b23SMartin Matuska.
9903ff01b23SMartin Matuska.It Sy zfs_disable_ivset_guid_check Ns = Ns Sy 0 Ns | Ns 1 Pq int
9913ff01b23SMartin MatuskaDisables requirement for IVset GUIDs to be present and match when doing a raw
9923ff01b23SMartin Matuskareceive of encrypted datasets.
9933ff01b23SMartin MatuskaIntended for users whose pools were created with
9943ff01b23SMartin MatuskaOpenZFS pre-release versions and now have compatibility issues.
9953ff01b23SMartin Matuska.
9963ff01b23SMartin Matuska.It Sy zfs_key_max_salt_uses Ns = Ns Sy 400000000 Po 4*10^8 Pc Pq ulong
9973ff01b23SMartin MatuskaMaximum number of uses of a single salt value before generating a new one for
9983ff01b23SMartin Matuskaencrypted datasets.
9993ff01b23SMartin MatuskaThe default value is also the maximum.
10003ff01b23SMartin Matuska.
10013ff01b23SMartin Matuska.It Sy zfs_object_mutex_size Ns = Ns Sy 64 Pq uint
10023ff01b23SMartin MatuskaSize of the znode hashtable used for holds.
10033ff01b23SMartin Matuska.Pp
10043ff01b23SMartin MatuskaDue to the need to hold locks on objects that may not exist yet, kernel mutexes
10053ff01b23SMartin Matuskaare not created per-object and instead a hashtable is used where collisions
10063ff01b23SMartin Matuskawill result in objects waiting when there is not actually contention on the
10073ff01b23SMartin Matuskasame object.
10083ff01b23SMartin Matuska.
10093ff01b23SMartin Matuska.It Sy zfs_slow_io_events_per_second Ns = Ns Sy 20 Ns /s Pq int
1010bb2d13b6SMartin MatuskaRate limit delay and deadman zevents (which report slow I/O operations) to this
1011bb2d13b6SMartin Matuskamany per
10123ff01b23SMartin Matuskasecond.
10133ff01b23SMartin Matuska.
1014dbd5678dSMartin Matuska.It Sy zfs_unflushed_max_mem_amt Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq u64
10153ff01b23SMartin MatuskaUpper-bound limit for unflushed metadata changes to be held by the
10163ff01b23SMartin Matuskalog spacemap in memory, in bytes.
10173ff01b23SMartin Matuska.
1018dbd5678dSMartin Matuska.It Sy zfs_unflushed_max_mem_ppm Ns = Ns Sy 1000 Ns ppm Po 0.1% Pc Pq u64
10193ff01b23SMartin MatuskaPart of overall system memory that ZFS allows to be used
10203ff01b23SMartin Matuskafor unflushed metadata changes by the log spacemap, in millionths.
10213ff01b23SMartin Matuska.
1022dbd5678dSMartin Matuska.It Sy zfs_unflushed_log_block_max Ns = Ns Sy 131072 Po 128k Pc Pq u64
10233ff01b23SMartin MatuskaDescribes the maximum number of log spacemap blocks allowed for each pool.
10243ff01b23SMartin MatuskaThe default value means that the space in all the log spacemaps
10253ff01b23SMartin Matuskacan add up to no more than
1026716fd348SMartin Matuska.Sy 131072
10273ff01b23SMartin Matuskablocks (which means
1028716fd348SMartin Matuska.Em 16 GiB
10293ff01b23SMartin Matuskaof logical space before compression and ditto blocks,
10303ff01b23SMartin Matuskaassuming that blocksize is
1031716fd348SMartin Matuska.Em 128 KiB ) .
10323ff01b23SMartin Matuska.Pp
10333ff01b23SMartin MatuskaThis tunable is important because it involves a trade-off between import
10343ff01b23SMartin Matuskatime after an unclean export and the frequency of flushing metaslabs.
10353ff01b23SMartin MatuskaThe higher this number is, the more log blocks we allow when the pool is
10363ff01b23SMartin Matuskaactive which means that we flush metaslabs less often and thus decrease
1037c03c5b1cSMartin Matuskathe number of I/O operations for spacemap updates per TXG.
10383ff01b23SMartin MatuskaAt the same time though, that means that in the event of an unclean export,
10393ff01b23SMartin Matuskathere will be more log spacemap blocks for us to read, inducing overhead
10403ff01b23SMartin Matuskain the import time of the pool.
10413ff01b23SMartin MatuskaThe lower the number, the amount of flushing increases, destroying log
10423ff01b23SMartin Matuskablocks quicker as they become obsolete faster, which leaves less blocks
10433ff01b23SMartin Matuskato be read during import time after a crash.
10443ff01b23SMartin Matuska.Pp
10453ff01b23SMartin MatuskaEach log spacemap block existing during pool import leads to approximately
10463ff01b23SMartin Matuskaone extra logical I/O issued.
10473ff01b23SMartin MatuskaThis is the reason why this tunable is exposed in terms of blocks rather
10483ff01b23SMartin Matuskathan space used.
10493ff01b23SMartin Matuska.
1050dbd5678dSMartin Matuska.It Sy zfs_unflushed_log_block_min Ns = Ns Sy 1000 Pq u64
10513ff01b23SMartin MatuskaIf the number of metaslabs is small and our incoming rate is high,
10523ff01b23SMartin Matuskawe could get into a situation that we are flushing all our metaslabs every TXG.
10533ff01b23SMartin MatuskaThus we always allow at least this many log blocks.
10543ff01b23SMartin Matuska.
1055dbd5678dSMartin Matuska.It Sy zfs_unflushed_log_block_pct Ns = Ns Sy 400 Ns % Pq u64
10563ff01b23SMartin MatuskaTunable used to determine the number of blocks that can be used for
10573ff01b23SMartin Matuskathe spacemap log, expressed as a percentage of the total number of
1058716fd348SMartin Matuskaunflushed metaslabs in the pool.
1059716fd348SMartin Matuska.
1060dbd5678dSMartin Matuska.It Sy zfs_unflushed_log_txg_max Ns = Ns Sy 1000 Pq u64
1061716fd348SMartin MatuskaTunable limiting maximum time in TXGs any metaslab may remain unflushed.
1062716fd348SMartin MatuskaIt effectively limits maximum number of unflushed per-TXG spacemap logs
1063716fd348SMartin Matuskathat need to be read after unclean pool export.
10643ff01b23SMartin Matuska.
10653ff01b23SMartin Matuska.It Sy zfs_unlink_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq uint
10663ff01b23SMartin MatuskaWhen enabled, files will not be asynchronously removed from the list of pending
10673ff01b23SMartin Matuskaunlinks and the space they consume will be leaked.
10683ff01b23SMartin MatuskaOnce this option has been disabled and the dataset is remounted,
10693ff01b23SMartin Matuskathe pending unlinks will be processed and the freed space returned to the pool.
10703ff01b23SMartin MatuskaThis option is used by the test suite.
10713ff01b23SMartin Matuska.
10723ff01b23SMartin Matuska.It Sy zfs_delete_blocks Ns = Ns Sy 20480 Pq ulong
10733ff01b23SMartin MatuskaThis is the used to define a large file for the purposes of deletion.
10743ff01b23SMartin MatuskaFiles containing more than
10753ff01b23SMartin Matuska.Sy zfs_delete_blocks
10763ff01b23SMartin Matuskawill be deleted asynchronously, while smaller files are deleted synchronously.
10773ff01b23SMartin MatuskaDecreasing this value will reduce the time spent in an
10783ff01b23SMartin Matuska.Xr unlink 2
1079bb2d13b6SMartin Matuskasystem call, at the expense of a longer delay before the freed space is
1080bb2d13b6SMartin Matuskaavailable.
1081dbd5678dSMartin MatuskaThis only applies on Linux.
10823ff01b23SMartin Matuska.
10833ff01b23SMartin Matuska.It Sy zfs_dirty_data_max Ns = Pq int
10843ff01b23SMartin MatuskaDetermines the dirty space limit in bytes.
10853ff01b23SMartin MatuskaOnce this limit is exceeded, new writes are halted until space frees up.
10863ff01b23SMartin MatuskaThis parameter takes precedence over
10873ff01b23SMartin Matuska.Sy zfs_dirty_data_max_percent .
10883ff01b23SMartin Matuska.No See Sx ZFS TRANSACTION DELAY .
10893ff01b23SMartin Matuska.Pp
10903ff01b23SMartin MatuskaDefaults to
10913ff01b23SMartin Matuska.Sy physical_ram/10 ,
10923ff01b23SMartin Matuskacapped at
10933ff01b23SMartin Matuska.Sy zfs_dirty_data_max_max .
10943ff01b23SMartin Matuska.
10953ff01b23SMartin Matuska.It Sy zfs_dirty_data_max_max Ns = Pq int
10963ff01b23SMartin MatuskaMaximum allowable value of
10973ff01b23SMartin Matuska.Sy zfs_dirty_data_max ,
10983ff01b23SMartin Matuskaexpressed in bytes.
10993ff01b23SMartin MatuskaThis limit is only enforced at module load time, and will be ignored if
11003ff01b23SMartin Matuska.Sy zfs_dirty_data_max
11013ff01b23SMartin Matuskais later changed.
11023ff01b23SMartin MatuskaThis parameter takes precedence over
11033ff01b23SMartin Matuska.Sy zfs_dirty_data_max_max_percent .
11043ff01b23SMartin Matuska.No See Sx ZFS TRANSACTION DELAY .
11053ff01b23SMartin Matuska.Pp
11063ff01b23SMartin MatuskaDefaults to
1107*15f0b8c3SMartin Matuska.Sy min(physical_ram/4, 4GiB) ,
1108*15f0b8c3SMartin Matuskaor
1109*15f0b8c3SMartin Matuska.Sy min(physical_ram/4, 1GiB)
1110*15f0b8c3SMartin Matuskafor 32-bit systems.
11113ff01b23SMartin Matuska.
1112be181ee2SMartin Matuska.It Sy zfs_dirty_data_max_max_percent Ns = Ns Sy 25 Ns % Pq uint
11133ff01b23SMartin MatuskaMaximum allowable value of
11143ff01b23SMartin Matuska.Sy zfs_dirty_data_max ,
11153ff01b23SMartin Matuskaexpressed as a percentage of physical RAM.
11163ff01b23SMartin MatuskaThis limit is only enforced at module load time, and will be ignored if
11173ff01b23SMartin Matuska.Sy zfs_dirty_data_max
11183ff01b23SMartin Matuskais later changed.
11193ff01b23SMartin MatuskaThe parameter
11203ff01b23SMartin Matuska.Sy zfs_dirty_data_max_max
11213ff01b23SMartin Matuskatakes precedence over this one.
11223ff01b23SMartin Matuska.No See Sx ZFS TRANSACTION DELAY .
11233ff01b23SMartin Matuska.
1124be181ee2SMartin Matuska.It Sy zfs_dirty_data_max_percent Ns = Ns Sy 10 Ns % Pq uint
11253ff01b23SMartin MatuskaDetermines the dirty space limit, expressed as a percentage of all memory.
11263ff01b23SMartin MatuskaOnce this limit is exceeded, new writes are halted until space frees up.
11273ff01b23SMartin MatuskaThe parameter
11283ff01b23SMartin Matuska.Sy zfs_dirty_data_max
11293ff01b23SMartin Matuskatakes precedence over this one.
11303ff01b23SMartin Matuska.No See Sx ZFS TRANSACTION DELAY .
11313ff01b23SMartin Matuska.Pp
11323ff01b23SMartin MatuskaSubject to
11333ff01b23SMartin Matuska.Sy zfs_dirty_data_max_max .
11343ff01b23SMartin Matuska.
1135be181ee2SMartin Matuska.It Sy zfs_dirty_data_sync_percent Ns = Ns Sy 20 Ns % Pq uint
11363ff01b23SMartin MatuskaStart syncing out a transaction group if there's at least this much dirty data
11373ff01b23SMartin Matuska.Pq as a percentage of Sy zfs_dirty_data_max .
11383ff01b23SMartin MatuskaThis should be less than
11393ff01b23SMartin Matuska.Sy zfs_vdev_async_write_active_min_dirty_percent .
11403ff01b23SMartin Matuska.
11413f9d360cSMartin Matuska.It Sy zfs_wrlog_data_max Ns = Pq int
11423f9d360cSMartin MatuskaThe upper limit of write-transaction zil log data size in bytes.
1143e3aa18adSMartin MatuskaWrite operations are throttled when approaching the limit until log data is
1144e3aa18adSMartin Matuskacleared out after transaction group sync.
1145e3aa18adSMartin MatuskaBecause of some overhead, it should be set at least 2 times the size of
11463f9d360cSMartin Matuska.Sy zfs_dirty_data_max
11473f9d360cSMartin Matuska.No to prevent harming normal write throughput .
11483f9d360cSMartin MatuskaIt also should be smaller than the size of the slog device if slog is present.
11493f9d360cSMartin Matuska.Pp
11503f9d360cSMartin MatuskaDefaults to
11513f9d360cSMartin Matuska.Sy zfs_dirty_data_max*2
11523f9d360cSMartin Matuska.
11533ff01b23SMartin Matuska.It Sy zfs_fallocate_reserve_percent Ns = Ns Sy 110 Ns % Pq uint
11543ff01b23SMartin MatuskaSince ZFS is a copy-on-write filesystem with snapshots, blocks cannot be
11553ff01b23SMartin Matuskapreallocated for a file in order to guarantee that later writes will not
11563ff01b23SMartin Matuskarun out of space.
11573ff01b23SMartin MatuskaInstead,
11583ff01b23SMartin Matuska.Xr fallocate 2
11593ff01b23SMartin Matuskaspace preallocation only checks that sufficient space is currently available
11603ff01b23SMartin Matuskain the pool or the user's project quota allocation,
11613ff01b23SMartin Matuskaand then creates a sparse file of the requested size.
11623ff01b23SMartin MatuskaThe requested space is multiplied by
11633ff01b23SMartin Matuska.Sy zfs_fallocate_reserve_percent
11643ff01b23SMartin Matuskato allow additional space for indirect blocks and other internal metadata.
11653ff01b23SMartin MatuskaSetting this to
11663ff01b23SMartin Matuska.Sy 0
11673ff01b23SMartin Matuskadisables support for
11683ff01b23SMartin Matuska.Xr fallocate 2
11693ff01b23SMartin Matuskaand causes it to return
11703ff01b23SMartin Matuska.Sy EOPNOTSUPP .
11713ff01b23SMartin Matuska.
11723ff01b23SMartin Matuska.It Sy zfs_fletcher_4_impl Ns = Ns Sy fastest Pq string
11733ff01b23SMartin MatuskaSelect a fletcher 4 implementation.
11743ff01b23SMartin Matuska.Pp
11753ff01b23SMartin MatuskaSupported selectors are:
11763ff01b23SMartin Matuska.Sy fastest , scalar , sse2 , ssse3 , avx2 , avx512f , avx512bw ,
11773ff01b23SMartin Matuska.No and Sy aarch64_neon .
11783ff01b23SMartin MatuskaAll except
11793ff01b23SMartin Matuska.Sy fastest No and Sy scalar
11803ff01b23SMartin Matuskarequire instruction set extensions to be available,
11813ff01b23SMartin Matuskaand will only appear if ZFS detects that they are present at runtime.
11823ff01b23SMartin MatuskaIf multiple implementations of fletcher 4 are available, the
11833ff01b23SMartin Matuska.Sy fastest
11843ff01b23SMartin Matuskawill be chosen using a micro benchmark.
11853ff01b23SMartin MatuskaSelecting
11863ff01b23SMartin Matuska.Sy scalar
11873ff01b23SMartin Matuskaresults in the original CPU-based calculation being used.
11883ff01b23SMartin MatuskaSelecting any option other than
11893ff01b23SMartin Matuska.Sy fastest No or Sy scalar
11903ff01b23SMartin Matuskaresults in vector instructions
11913ff01b23SMartin Matuskafrom the respective CPU instruction set being used.
11923ff01b23SMartin Matuska.
1193c7046f76SMartin Matuska.It Sy zfs_blake3_impl Ns = Ns Sy fastest Pq string
1194c7046f76SMartin MatuskaSelect a BLAKE3 implementation.
1195c7046f76SMartin Matuska.Pp
1196c7046f76SMartin MatuskaSupported selectors are:
1197c7046f76SMartin Matuska.Sy cycle , fastest , generic , sse2 , sse41 , avx2 , avx512 .
1198c7046f76SMartin MatuskaAll except
1199c7046f76SMartin Matuska.Sy cycle , fastest No and Sy generic
1200c7046f76SMartin Matuskarequire instruction set extensions to be available,
1201c7046f76SMartin Matuskaand will only appear if ZFS detects that they are present at runtime.
1202c7046f76SMartin MatuskaIf multiple implementations of BLAKE3 are available, the
1203c7046f76SMartin Matuska.Sy fastest will be chosen using a micro benchmark. You can see the
1204c7046f76SMartin Matuskabenchmark results by reading this kstat file:
1205c7046f76SMartin Matuska.Pa /proc/spl/kstat/zfs/chksum_bench .
1206c7046f76SMartin Matuska.
12073ff01b23SMartin Matuska.It Sy zfs_free_bpobj_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
12083ff01b23SMartin MatuskaEnable/disable the processing of the free_bpobj object.
12093ff01b23SMartin Matuska.
1210dbd5678dSMartin Matuska.It Sy zfs_async_block_max_blocks Ns = Ns Sy UINT64_MAX Po unlimited Pc Pq u64
12113ff01b23SMartin MatuskaMaximum number of blocks freed in a single TXG.
12123ff01b23SMartin Matuska.
1213dbd5678dSMartin Matuska.It Sy zfs_max_async_dedup_frees Ns = Ns Sy 100000 Po 10^5 Pc Pq u64
12143ff01b23SMartin MatuskaMaximum number of dedup blocks freed in a single TXG.
12153ff01b23SMartin Matuska.
1216be181ee2SMartin Matuska.It Sy zfs_vdev_async_read_max_active Ns = Ns Sy 3 Pq uint
12173ff01b23SMartin MatuskaMaximum asynchronous read I/O operations active to each device.
12183ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12193ff01b23SMartin Matuska.
1220be181ee2SMartin Matuska.It Sy zfs_vdev_async_read_min_active Ns = Ns Sy 1 Pq uint
12213ff01b23SMartin MatuskaMinimum asynchronous read I/O operation active to each device.
12223ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12233ff01b23SMartin Matuska.
1224be181ee2SMartin Matuska.It Sy zfs_vdev_async_write_active_max_dirty_percent Ns = Ns Sy 60 Ns % Pq uint
12253ff01b23SMartin MatuskaWhen the pool has more than this much dirty data, use
12263ff01b23SMartin Matuska.Sy zfs_vdev_async_write_max_active
12273ff01b23SMartin Matuskato limit active async writes.
12283ff01b23SMartin MatuskaIf the dirty data is between the minimum and maximum,
12293ff01b23SMartin Matuskathe active I/O limit is linearly interpolated.
12303ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12313ff01b23SMartin Matuska.
1232be181ee2SMartin Matuska.It Sy zfs_vdev_async_write_active_min_dirty_percent Ns = Ns Sy 30 Ns % Pq uint
12333ff01b23SMartin MatuskaWhen the pool has less than this much dirty data, use
12343ff01b23SMartin Matuska.Sy zfs_vdev_async_write_min_active
12353ff01b23SMartin Matuskato limit active async writes.
12363ff01b23SMartin MatuskaIf the dirty data is between the minimum and maximum,
12373ff01b23SMartin Matuskathe active I/O limit is linearly
12383ff01b23SMartin Matuskainterpolated.
12393ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12403ff01b23SMartin Matuska.
1241bb2d13b6SMartin Matuska.It Sy zfs_vdev_async_write_max_active Ns = Ns Sy 10 Pq uint
12423ff01b23SMartin MatuskaMaximum asynchronous write I/O operations active to each device.
12433ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12443ff01b23SMartin Matuska.
1245be181ee2SMartin Matuska.It Sy zfs_vdev_async_write_min_active Ns = Ns Sy 2 Pq uint
12463ff01b23SMartin MatuskaMinimum asynchronous write I/O operations active to each device.
12473ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12483ff01b23SMartin Matuska.Pp
12493ff01b23SMartin MatuskaLower values are associated with better latency on rotational media but poorer
12503ff01b23SMartin Matuskaresilver performance.
12513ff01b23SMartin MatuskaThe default value of
12523ff01b23SMartin Matuska.Sy 2
12533ff01b23SMartin Matuskawas chosen as a compromise.
12543ff01b23SMartin MatuskaA value of
12553ff01b23SMartin Matuska.Sy 3
12563ff01b23SMartin Matuskahas been shown to improve resilver performance further at a cost of
12573ff01b23SMartin Matuskafurther increasing latency.
12583ff01b23SMartin Matuska.
1259be181ee2SMartin Matuska.It Sy zfs_vdev_initializing_max_active Ns = Ns Sy 1 Pq uint
12603ff01b23SMartin MatuskaMaximum initializing I/O operations active to each device.
12613ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12623ff01b23SMartin Matuska.
1263be181ee2SMartin Matuska.It Sy zfs_vdev_initializing_min_active Ns = Ns Sy 1 Pq uint
12643ff01b23SMartin MatuskaMinimum initializing I/O operations active to each device.
12653ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12663ff01b23SMartin Matuska.
1267be181ee2SMartin Matuska.It Sy zfs_vdev_max_active Ns = Ns Sy 1000 Pq uint
12683ff01b23SMartin MatuskaThe maximum number of I/O operations active to each device.
12693ff01b23SMartin MatuskaIdeally, this will be at least the sum of each queue's
12703ff01b23SMartin Matuska.Sy max_active .
12713ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12723ff01b23SMartin Matuska.
1273dbd5678dSMartin Matuska.It Sy zfs_vdev_open_timeout_ms Ns = Ns Sy 1000 Pq uint
1274dbd5678dSMartin MatuskaTimeout value to wait before determining a device is missing
1275dbd5678dSMartin Matuskaduring import.
1276dbd5678dSMartin MatuskaThis is helpful for transient missing paths due
1277dbd5678dSMartin Matuskato links being briefly removed and recreated in response to
1278dbd5678dSMartin Matuskaudev events.
1279dbd5678dSMartin Matuska.
1280be181ee2SMartin Matuska.It Sy zfs_vdev_rebuild_max_active Ns = Ns Sy 3 Pq uint
12813ff01b23SMartin MatuskaMaximum sequential resilver I/O operations active to each device.
12823ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12833ff01b23SMartin Matuska.
1284be181ee2SMartin Matuska.It Sy zfs_vdev_rebuild_min_active Ns = Ns Sy 1 Pq uint
12853ff01b23SMartin MatuskaMinimum sequential resilver I/O operations active to each device.
12863ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12873ff01b23SMartin Matuska.
1288be181ee2SMartin Matuska.It Sy zfs_vdev_removal_max_active Ns = Ns Sy 2 Pq uint
12893ff01b23SMartin MatuskaMaximum removal I/O operations active to each device.
12903ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12913ff01b23SMartin Matuska.
1292be181ee2SMartin Matuska.It Sy zfs_vdev_removal_min_active Ns = Ns Sy 1 Pq uint
12933ff01b23SMartin MatuskaMinimum removal I/O operations active to each device.
12943ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12953ff01b23SMartin Matuska.
1296be181ee2SMartin Matuska.It Sy zfs_vdev_scrub_max_active Ns = Ns Sy 2 Pq uint
12973ff01b23SMartin MatuskaMaximum scrub I/O operations active to each device.
12983ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
12993ff01b23SMartin Matuska.
1300be181ee2SMartin Matuska.It Sy zfs_vdev_scrub_min_active Ns = Ns Sy 1 Pq uint
13013ff01b23SMartin MatuskaMinimum scrub I/O operations active to each device.
13023ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
13033ff01b23SMartin Matuska.
1304be181ee2SMartin Matuska.It Sy zfs_vdev_sync_read_max_active Ns = Ns Sy 10 Pq uint
13053ff01b23SMartin MatuskaMaximum synchronous read I/O operations active to each device.
13063ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
13073ff01b23SMartin Matuska.
1308be181ee2SMartin Matuska.It Sy zfs_vdev_sync_read_min_active Ns = Ns Sy 10 Pq uint
13093ff01b23SMartin MatuskaMinimum synchronous read I/O operations active to each device.
13103ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
13113ff01b23SMartin Matuska.
1312be181ee2SMartin Matuska.It Sy zfs_vdev_sync_write_max_active Ns = Ns Sy 10 Pq uint
13133ff01b23SMartin MatuskaMaximum synchronous write I/O operations active to each device.
13143ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
13153ff01b23SMartin Matuska.
1316be181ee2SMartin Matuska.It Sy zfs_vdev_sync_write_min_active Ns = Ns Sy 10 Pq uint
13173ff01b23SMartin MatuskaMinimum synchronous write I/O operations active to each device.
13183ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
13193ff01b23SMartin Matuska.
1320be181ee2SMartin Matuska.It Sy zfs_vdev_trim_max_active Ns = Ns Sy 2 Pq uint
13213ff01b23SMartin MatuskaMaximum trim/discard I/O operations active to each device.
13223ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
13233ff01b23SMartin Matuska.
1324be181ee2SMartin Matuska.It Sy zfs_vdev_trim_min_active Ns = Ns Sy 1 Pq uint
13253ff01b23SMartin MatuskaMinimum trim/discard I/O operations active to each device.
13263ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
13273ff01b23SMartin Matuska.
1328be181ee2SMartin Matuska.It Sy zfs_vdev_nia_delay Ns = Ns Sy 5 Pq uint
13293ff01b23SMartin MatuskaFor non-interactive I/O (scrub, resilver, removal, initialize and rebuild),
13303ff01b23SMartin Matuskathe number of concurrently-active I/O operations is limited to
13313ff01b23SMartin Matuska.Sy zfs_*_min_active ,
13323ff01b23SMartin Matuskaunless the vdev is "idle".
1333e92ffd9bSMartin MatuskaWhen there are no interactive I/O operations active (synchronous or otherwise),
13343ff01b23SMartin Matuskaand
13353ff01b23SMartin Matuska.Sy zfs_vdev_nia_delay
13363ff01b23SMartin Matuskaoperations have completed since the last interactive operation,
13373ff01b23SMartin Matuskathen the vdev is considered to be "idle",
13383ff01b23SMartin Matuskaand the number of concurrently-active non-interactive operations is increased to
13393ff01b23SMartin Matuska.Sy zfs_*_max_active .
13403ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
13413ff01b23SMartin Matuska.
1342be181ee2SMartin Matuska.It Sy zfs_vdev_nia_credit Ns = Ns Sy 5 Pq uint
13433ff01b23SMartin MatuskaSome HDDs tend to prioritize sequential I/O so strongly, that concurrent
13443ff01b23SMartin Matuskarandom I/O latency reaches several seconds.
13453ff01b23SMartin MatuskaOn some HDDs this happens even if sequential I/O operations
13463ff01b23SMartin Matuskaare submitted one at a time, and so setting
13473ff01b23SMartin Matuska.Sy zfs_*_max_active Ns = Sy 1
13483ff01b23SMartin Matuskadoes not help.
13493ff01b23SMartin MatuskaTo prevent non-interactive I/O, like scrub,
13503ff01b23SMartin Matuskafrom monopolizing the device, no more than
13513ff01b23SMartin Matuska.Sy zfs_vdev_nia_credit operations can be sent
13523ff01b23SMartin Matuskawhile there are outstanding incomplete interactive operations.
13533ff01b23SMartin MatuskaThis enforced wait ensures the HDD services the interactive I/O
13543ff01b23SMartin Matuskawithin a reasonable amount of time.
13553ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
13563ff01b23SMartin Matuska.
1357be181ee2SMartin Matuska.It Sy zfs_vdev_queue_depth_pct Ns = Ns Sy 1000 Ns % Pq uint
13583ff01b23SMartin MatuskaMaximum number of queued allocations per top-level vdev expressed as
13593ff01b23SMartin Matuskaa percentage of
13603ff01b23SMartin Matuska.Sy zfs_vdev_async_write_max_active ,
13613ff01b23SMartin Matuskawhich allows the system to detect devices that are more capable
13623ff01b23SMartin Matuskaof handling allocations and to allocate more blocks to those devices.
13633ff01b23SMartin MatuskaThis allows for dynamic allocation distribution when devices are imbalanced,
13643ff01b23SMartin Matuskaas fuller devices will tend to be slower than empty devices.
13653ff01b23SMartin Matuska.Pp
13663ff01b23SMartin MatuskaAlso see
13673ff01b23SMartin Matuska.Sy zio_dva_throttle_enabled .
13683ff01b23SMartin Matuska.
1369dbd5678dSMartin Matuska.It Sy zfs_vdev_failfast_mask Ns = Ns Sy 1 Pq uint
1370dbd5678dSMartin MatuskaDefines if the driver should retire on a given error type.
1371dbd5678dSMartin MatuskaThe following options may be bitwise-ored together:
1372dbd5678dSMartin Matuska.TS
1373dbd5678dSMartin Matuskabox;
1374dbd5678dSMartin Matuskalbz r l l .
1375dbd5678dSMartin Matuska	Value	Name	Description
1376dbd5678dSMartin Matuska_
1377dbd5678dSMartin Matuska	1	Device	No driver retries on device errors
1378dbd5678dSMartin Matuska	2	Transport	No driver retries on transport errors.
1379dbd5678dSMartin Matuska	4	Driver	No driver retries on driver errors.
1380dbd5678dSMartin Matuska.TE
1381dbd5678dSMartin Matuska.
13823ff01b23SMartin Matuska.It Sy zfs_expire_snapshot Ns = Ns Sy 300 Ns s Pq int
13833ff01b23SMartin MatuskaTime before expiring
13843ff01b23SMartin Matuska.Pa .zfs/snapshot .
13853ff01b23SMartin Matuska.
13863ff01b23SMartin Matuska.It Sy zfs_admin_snapshot Ns = Ns Sy 0 Ns | Ns 1 Pq int
13873ff01b23SMartin MatuskaAllow the creation, removal, or renaming of entries in the
13883ff01b23SMartin Matuska.Sy .zfs/snapshot
13893ff01b23SMartin Matuskadirectory to cause the creation, destruction, or renaming of snapshots.
13903ff01b23SMartin MatuskaWhen enabled, this functionality works both locally and over NFS exports
13913ff01b23SMartin Matuskawhich have the
13923ff01b23SMartin Matuska.Em no_root_squash
13933ff01b23SMartin Matuskaoption set.
13943ff01b23SMartin Matuska.
13953ff01b23SMartin Matuska.It Sy zfs_flags Ns = Ns Sy 0 Pq int
13963ff01b23SMartin MatuskaSet additional debugging flags.
13973ff01b23SMartin MatuskaThe following flags may be bitwise-ored together:
13983ff01b23SMartin Matuska.TS
13993ff01b23SMartin Matuskabox;
14003ff01b23SMartin Matuskalbz r l l .
1401dbd5678dSMartin Matuska	Value	Name	Description
14023ff01b23SMartin Matuska_
14033ff01b23SMartin Matuska	1	ZFS_DEBUG_DPRINTF	Enable dprintf entries in the debug log.
14043ff01b23SMartin Matuska*	2	ZFS_DEBUG_DBUF_VERIFY	Enable extra dbuf verifications.
14053ff01b23SMartin Matuska*	4	ZFS_DEBUG_DNODE_VERIFY	Enable extra dnode verifications.
14063ff01b23SMartin Matuska	8	ZFS_DEBUG_SNAPNAMES	Enable snapshot name verification.
1407*15f0b8c3SMartin Matuska*	16	ZFS_DEBUG_MODIFY	Check for illegally modified ARC buffers.
14083ff01b23SMartin Matuska	64	ZFS_DEBUG_ZIO_FREE	Enable verification of block frees.
14093ff01b23SMartin Matuska	128	ZFS_DEBUG_HISTOGRAM_VERIFY	Enable extra spacemap histogram verifications.
14103ff01b23SMartin Matuska	256	ZFS_DEBUG_METASLAB_VERIFY	Verify space accounting on disk matches in-memory \fBrange_trees\fP.
14113ff01b23SMartin Matuska	512	ZFS_DEBUG_SET_ERROR	Enable \fBSET_ERROR\fP and dprintf entries in the debug log.
14123ff01b23SMartin Matuska	1024	ZFS_DEBUG_INDIRECT_REMAP	Verify split blocks created by device removal.
14133ff01b23SMartin Matuska	2048	ZFS_DEBUG_TRIM	Verify TRIM ranges are always within the allocatable range tree.
14143ff01b23SMartin Matuska	4096	ZFS_DEBUG_LOG_SPACEMAP	Verify that the log summary is consistent with the spacemap log
14153ff01b23SMartin Matuska			       and enable \fBzfs_dbgmsgs\fP for metaslab loading and flushing.
14163ff01b23SMartin Matuska.TE
14173ff01b23SMartin Matuska.Sy \& * No Requires debug build .
14183ff01b23SMartin Matuska.
1419c7046f76SMartin Matuska.It Sy zfs_btree_verify_intensity Ns = Ns Sy 0 Pq uint
1420c7046f76SMartin MatuskaEnables btree verification.
1421c7046f76SMartin MatuskaThe following settings are culminative:
1422c7046f76SMartin Matuska.TS
1423c7046f76SMartin Matuskabox;
1424c7046f76SMartin Matuskalbz r l l .
1425c7046f76SMartin Matuska	Value	Description
1426c7046f76SMartin Matuska
1427c7046f76SMartin Matuska	1	Verify height.
1428c7046f76SMartin Matuska	2	Verify pointers from children to parent.
1429c7046f76SMartin Matuska	3	Verify element counts.
1430c7046f76SMartin Matuska	4	Verify element order. (expensive)
1431c7046f76SMartin Matuska*	5	Verify unused memory is poisoned. (expensive)
1432c7046f76SMartin Matuska.TE
1433c7046f76SMartin Matuska.Sy \& * No Requires debug build .
1434c7046f76SMartin Matuska.
14353ff01b23SMartin Matuska.It Sy zfs_free_leak_on_eio Ns = Ns Sy 0 Ns | Ns 1 Pq int
14363ff01b23SMartin MatuskaIf destroy encounters an
14373ff01b23SMartin Matuska.Sy EIO
14383ff01b23SMartin Matuskawhile reading metadata (e.g. indirect blocks),
14393ff01b23SMartin Matuskaspace referenced by the missing metadata can not be freed.
14403ff01b23SMartin MatuskaNormally this causes the background destroy to become "stalled",
14413ff01b23SMartin Matuskaas it is unable to make forward progress.
14423ff01b23SMartin MatuskaWhile in this stalled state, all remaining space to free
14433ff01b23SMartin Matuskafrom the error-encountering filesystem is "temporarily leaked".
14443ff01b23SMartin MatuskaSet this flag to cause it to ignore the
14453ff01b23SMartin Matuska.Sy EIO ,
14463ff01b23SMartin Matuskapermanently leak the space from indirect blocks that can not be read,
14473ff01b23SMartin Matuskaand continue to free everything else that it can.
14483ff01b23SMartin Matuska.Pp
14493ff01b23SMartin MatuskaThe default "stalling" behavior is useful if the storage partially
14503ff01b23SMartin Matuskafails (i.e. some but not all I/O operations fail), and then later recovers.
14513ff01b23SMartin MatuskaIn this case, we will be able to continue pool operations while it is
14523ff01b23SMartin Matuskapartially failed, and when it recovers, we can continue to free the
14533ff01b23SMartin Matuskaspace, with no leaks.
14543ff01b23SMartin MatuskaNote, however, that this case is actually fairly rare.
14553ff01b23SMartin Matuska.Pp
14563ff01b23SMartin MatuskaTypically pools either
14573ff01b23SMartin Matuska.Bl -enum -compact -offset 4n -width "1."
14583ff01b23SMartin Matuska.It
14593ff01b23SMartin Matuskafail completely (but perhaps temporarily,
14603ff01b23SMartin Matuskae.g. due to a top-level vdev going offline), or
14613ff01b23SMartin Matuska.It
14623ff01b23SMartin Matuskahave localized, permanent errors (e.g. disk returns the wrong data
14633ff01b23SMartin Matuskadue to bit flip or firmware bug).
14643ff01b23SMartin Matuska.El
14653ff01b23SMartin MatuskaIn the former case, this setting does not matter because the
14663ff01b23SMartin Matuskapool will be suspended and the sync thread will not be able to make
14673ff01b23SMartin Matuskaforward progress regardless.
14683ff01b23SMartin MatuskaIn the latter, because the error is permanent, the best we can do
14693ff01b23SMartin Matuskais leak the minimum amount of space,
14703ff01b23SMartin Matuskawhich is what setting this flag will do.
14713ff01b23SMartin MatuskaIt is therefore reasonable for this flag to normally be set,
14723ff01b23SMartin Matuskabut we chose the more conservative approach of not setting it,
14733ff01b23SMartin Matuskaso that there is no possibility of
14743ff01b23SMartin Matuskaleaking space in the "partial temporary" failure case.
14753ff01b23SMartin Matuska.
1476be181ee2SMartin Matuska.It Sy zfs_free_min_time_ms Ns = Ns Sy 1000 Ns ms Po 1s Pc Pq uint
14773ff01b23SMartin MatuskaDuring a
14783ff01b23SMartin Matuska.Nm zfs Cm destroy
14793ff01b23SMartin Matuskaoperation using the
14803ff01b23SMartin Matuska.Sy async_destroy
14813ff01b23SMartin Matuskafeature,
14823ff01b23SMartin Matuskaa minimum of this much time will be spent working on freeing blocks per TXG.
14833ff01b23SMartin Matuska.
1484be181ee2SMartin Matuska.It Sy zfs_obsolete_min_time_ms Ns = Ns Sy 500 Ns ms Pq uint
14853ff01b23SMartin MatuskaSimilar to
14863ff01b23SMartin Matuska.Sy zfs_free_min_time_ms ,
14873ff01b23SMartin Matuskabut for cleanup of old indirection records for removed vdevs.
14883ff01b23SMartin Matuska.
1489dbd5678dSMartin Matuska.It Sy zfs_immediate_write_sz Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq s64
14903ff01b23SMartin MatuskaLargest data block to write to the ZIL.
14913ff01b23SMartin MatuskaLarger blocks will be treated as if the dataset being written to had the
14923ff01b23SMartin Matuska.Sy logbias Ns = Ns Sy throughput
14933ff01b23SMartin Matuskaproperty set.
14943ff01b23SMartin Matuska.
1495dbd5678dSMartin Matuska.It Sy zfs_initialize_value Ns = Ns Sy 16045690984833335022 Po 0xDEADBEEFDEADBEEE Pc Pq u64
14963ff01b23SMartin MatuskaPattern written to vdev free space by
14973ff01b23SMartin Matuska.Xr zpool-initialize 8 .
14983ff01b23SMartin Matuska.
1499dbd5678dSMartin Matuska.It Sy zfs_initialize_chunk_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64
15003ff01b23SMartin MatuskaSize of writes used by
15013ff01b23SMartin Matuska.Xr zpool-initialize 8 .
15023ff01b23SMartin MatuskaThis option is used by the test suite.
15033ff01b23SMartin Matuska.
1504dbd5678dSMartin Matuska.It Sy zfs_livelist_max_entries Ns = Ns Sy 500000 Po 5*10^5 Pc Pq u64
15053ff01b23SMartin MatuskaThe threshold size (in block pointers) at which we create a new sub-livelist.
15063ff01b23SMartin MatuskaLarger sublists are more costly from a memory perspective but the fewer
15073ff01b23SMartin Matuskasublists there are, the lower the cost of insertion.
15083ff01b23SMartin Matuska.
15093ff01b23SMartin Matuska.It Sy zfs_livelist_min_percent_shared Ns = Ns Sy 75 Ns % Pq int
15103ff01b23SMartin MatuskaIf the amount of shared space between a snapshot and its clone drops below
15113ff01b23SMartin Matuskathis threshold, the clone turns off the livelist and reverts to the old
15123ff01b23SMartin Matuskadeletion method.
15133ff01b23SMartin MatuskaThis is in place because livelists no long give us a benefit
15143ff01b23SMartin Matuskaonce a clone has been overwritten enough.
15153ff01b23SMartin Matuska.
15163ff01b23SMartin Matuska.It Sy zfs_livelist_condense_new_alloc Ns = Ns Sy 0 Pq int
15173ff01b23SMartin MatuskaIncremented each time an extra ALLOC blkptr is added to a livelist entry while
15183ff01b23SMartin Matuskait is being condensed.
15193ff01b23SMartin MatuskaThis option is used by the test suite to track race conditions.
15203ff01b23SMartin Matuska.
15213ff01b23SMartin Matuska.It Sy zfs_livelist_condense_sync_cancel Ns = Ns Sy 0 Pq int
15223ff01b23SMartin MatuskaIncremented each time livelist condensing is canceled while in
15233ff01b23SMartin Matuska.Fn spa_livelist_condense_sync .
15243ff01b23SMartin MatuskaThis option is used by the test suite to track race conditions.
15253ff01b23SMartin Matuska.
15263ff01b23SMartin Matuska.It Sy zfs_livelist_condense_sync_pause Ns = Ns Sy 0 Ns | Ns 1 Pq int
15273ff01b23SMartin MatuskaWhen set, the livelist condense process pauses indefinitely before
1528e92ffd9bSMartin Matuskaexecuting the synctask \(em
15293ff01b23SMartin Matuska.Fn spa_livelist_condense_sync .
15303ff01b23SMartin MatuskaThis option is used by the test suite to trigger race conditions.
15313ff01b23SMartin Matuska.
15323ff01b23SMartin Matuska.It Sy zfs_livelist_condense_zthr_cancel Ns = Ns Sy 0 Pq int
15333ff01b23SMartin MatuskaIncremented each time livelist condensing is canceled while in
15343ff01b23SMartin Matuska.Fn spa_livelist_condense_cb .
15353ff01b23SMartin MatuskaThis option is used by the test suite to track race conditions.
15363ff01b23SMartin Matuska.
15373ff01b23SMartin Matuska.It Sy zfs_livelist_condense_zthr_pause Ns = Ns Sy 0 Ns | Ns 1 Pq int
15383ff01b23SMartin MatuskaWhen set, the livelist condense process pauses indefinitely before
15393ff01b23SMartin Matuskaexecuting the open context condensing work in
15403ff01b23SMartin Matuska.Fn spa_livelist_condense_cb .
15413ff01b23SMartin MatuskaThis option is used by the test suite to trigger race conditions.
15423ff01b23SMartin Matuska.
1543dbd5678dSMartin Matuska.It Sy zfs_lua_max_instrlimit Ns = Ns Sy 100000000 Po 10^8 Pc Pq u64
15443ff01b23SMartin MatuskaThe maximum execution time limit that can be set for a ZFS channel program,
15453ff01b23SMartin Matuskaspecified as a number of Lua instructions.
15463ff01b23SMartin Matuska.
1547dbd5678dSMartin Matuska.It Sy zfs_lua_max_memlimit Ns = Ns Sy 104857600 Po 100 MiB Pc Pq u64
15483ff01b23SMartin MatuskaThe maximum memory limit that can be set for a ZFS channel program, specified
15493ff01b23SMartin Matuskain bytes.
15503ff01b23SMartin Matuska.
15513ff01b23SMartin Matuska.It Sy zfs_max_dataset_nesting Ns = Ns Sy 50 Pq int
15523ff01b23SMartin MatuskaThe maximum depth of nested datasets.
15533ff01b23SMartin MatuskaThis value can be tuned temporarily to
15543ff01b23SMartin Matuskafix existing datasets that exceed the predefined limit.
15553ff01b23SMartin Matuska.
1556dbd5678dSMartin Matuska.It Sy zfs_max_log_walking Ns = Ns Sy 5 Pq u64
15573ff01b23SMartin MatuskaThe number of past TXGs that the flushing algorithm of the log spacemap
15583ff01b23SMartin Matuskafeature uses to estimate incoming log blocks.
15593ff01b23SMartin Matuska.
1560dbd5678dSMartin Matuska.It Sy zfs_max_logsm_summary_length Ns = Ns Sy 10 Pq u64
15613ff01b23SMartin MatuskaMaximum number of rows allowed in the summary of the spacemap log.
15623ff01b23SMartin Matuska.
1563be181ee2SMartin Matuska.It Sy zfs_max_recordsize Ns = Ns Sy 16777216 Po 16 MiB Pc Pq uint
15643ff01b23SMartin MatuskaWe currently support block sizes from
1565716fd348SMartin Matuska.Em 512 Po 512 B Pc No to Em 16777216 Po 16 MiB Pc .
15663ff01b23SMartin MatuskaThe benefits of larger blocks, and thus larger I/O,
15673ff01b23SMartin Matuskaneed to be weighed against the cost of COWing a giant block to modify one byte.
15683ff01b23SMartin MatuskaAdditionally, very large blocks can have an impact on I/O latency,
15693ff01b23SMartin Matuskaand also potentially on the memory allocator.
1570716fd348SMartin MatuskaTherefore, we formerly forbade creating blocks larger than 1M.
1571716fd348SMartin MatuskaLarger blocks could be created by changing it,
15723ff01b23SMartin Matuskaand pools with larger blocks can always be imported and used,
15733ff01b23SMartin Matuskaregardless of this setting.
15743ff01b23SMartin Matuska.
15753ff01b23SMartin Matuska.It Sy zfs_allow_redacted_dataset_mount Ns = Ns Sy 0 Ns | Ns 1 Pq int
15763ff01b23SMartin MatuskaAllow datasets received with redacted send/receive to be mounted.
15773ff01b23SMartin MatuskaNormally disabled because these datasets may be missing key data.
15783ff01b23SMartin Matuska.
1579dbd5678dSMartin Matuska.It Sy zfs_min_metaslabs_to_flush Ns = Ns Sy 1 Pq u64
15803ff01b23SMartin MatuskaMinimum number of metaslabs to flush per dirty TXG.
15813ff01b23SMartin Matuska.
1582be181ee2SMartin Matuska.It Sy zfs_metaslab_fragmentation_threshold Ns = Ns Sy 70 Ns % Pq uint
15833ff01b23SMartin MatuskaAllow metaslabs to keep their active state as long as their fragmentation
15843ff01b23SMartin Matuskapercentage is no more than this value.
15853ff01b23SMartin MatuskaAn active metaslab that exceeds this threshold
15863ff01b23SMartin Matuskawill no longer keep its active status allowing better metaslabs to be selected.
15873ff01b23SMartin Matuska.
1588be181ee2SMartin Matuska.It Sy zfs_mg_fragmentation_threshold Ns = Ns Sy 95 Ns % Pq uint
15893ff01b23SMartin MatuskaMetaslab groups are considered eligible for allocations if their
15903ff01b23SMartin Matuskafragmentation metric (measured as a percentage) is less than or equal to
15913ff01b23SMartin Matuskathis value.
15923ff01b23SMartin MatuskaIf a metaslab group exceeds this threshold then it will be
15933ff01b23SMartin Matuskaskipped unless all metaslab groups within the metaslab class have also
15943ff01b23SMartin Matuskacrossed this threshold.
15953ff01b23SMartin Matuska.
1596be181ee2SMartin Matuska.It Sy zfs_mg_noalloc_threshold Ns = Ns Sy 0 Ns % Pq uint
15973ff01b23SMartin MatuskaDefines a threshold at which metaslab groups should be eligible for allocations.
15983ff01b23SMartin MatuskaThe value is expressed as a percentage of free space
15993ff01b23SMartin Matuskabeyond which a metaslab group is always eligible for allocations.
16003ff01b23SMartin MatuskaIf a metaslab group's free space is less than or equal to the
16013ff01b23SMartin Matuskathreshold, the allocator will avoid allocating to that group
16023ff01b23SMartin Matuskaunless all groups in the pool have reached the threshold.
16033ff01b23SMartin MatuskaOnce all groups have reached the threshold, all groups are allowed to accept
16043ff01b23SMartin Matuskaallocations.
16053ff01b23SMartin MatuskaThe default value of
16063ff01b23SMartin Matuska.Sy 0
1607bb2d13b6SMartin Matuskadisables the feature and causes all metaslab groups to be eligible for
1608bb2d13b6SMartin Matuskaallocations.
16093ff01b23SMartin Matuska.Pp
16103ff01b23SMartin MatuskaThis parameter allows one to deal with pools having heavily imbalanced
16113ff01b23SMartin Matuskavdevs such as would be the case when a new vdev has been added.
16123ff01b23SMartin MatuskaSetting the threshold to a non-zero percentage will stop allocations
16133ff01b23SMartin Matuskafrom being made to vdevs that aren't filled to the specified percentage
16143ff01b23SMartin Matuskaand allow lesser filled vdevs to acquire more allocations than they
16153ff01b23SMartin Matuskaotherwise would under the old
16163ff01b23SMartin Matuska.Sy zfs_mg_alloc_failures
16173ff01b23SMartin Matuskafacility.
16183ff01b23SMartin Matuska.
16193ff01b23SMartin Matuska.It Sy zfs_ddt_data_is_special Ns = Ns Sy 1 Ns | Ns 0 Pq int
16203ff01b23SMartin MatuskaIf enabled, ZFS will place DDT data into the special allocation class.
16213ff01b23SMartin Matuska.
16223ff01b23SMartin Matuska.It Sy zfs_user_indirect_is_special Ns = Ns Sy 1 Ns | Ns 0 Pq int
16233ff01b23SMartin MatuskaIf enabled, ZFS will place user data indirect blocks
16243ff01b23SMartin Matuskainto the special allocation class.
16253ff01b23SMartin Matuska.
1626be181ee2SMartin Matuska.It Sy zfs_multihost_history Ns = Ns Sy 0 Pq uint
1627bb2d13b6SMartin MatuskaHistorical statistics for this many latest multihost updates will be available
1628bb2d13b6SMartin Matuskain
16293ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /multihost .
16303ff01b23SMartin Matuska.
1631dbd5678dSMartin Matuska.It Sy zfs_multihost_interval Ns = Ns Sy 1000 Ns ms Po 1 s Pc Pq u64
16323ff01b23SMartin MatuskaUsed to control the frequency of multihost writes which are performed when the
16333ff01b23SMartin Matuska.Sy multihost
16343ff01b23SMartin Matuskapool property is on.
16353ff01b23SMartin MatuskaThis is one of the factors used to determine the
16363ff01b23SMartin Matuskalength of the activity check during import.
16373ff01b23SMartin Matuska.Pp
16383ff01b23SMartin MatuskaThe multihost write period is
1639e92ffd9bSMartin Matuska.Sy zfs_multihost_interval No / Sy leaf-vdevs .
16403ff01b23SMartin MatuskaOn average a multihost write will be issued for each leaf vdev
16413ff01b23SMartin Matuskaevery
16423ff01b23SMartin Matuska.Sy zfs_multihost_interval
16433ff01b23SMartin Matuskamilliseconds.
16443ff01b23SMartin MatuskaIn practice, the observed period can vary with the I/O load
16453ff01b23SMartin Matuskaand this observed value is the delay which is stored in the uberblock.
16463ff01b23SMartin Matuska.
16473ff01b23SMartin Matuska.It Sy zfs_multihost_import_intervals Ns = Ns Sy 20 Pq uint
16483ff01b23SMartin MatuskaUsed to control the duration of the activity test on import.
16493ff01b23SMartin MatuskaSmaller values of
16503ff01b23SMartin Matuska.Sy zfs_multihost_import_intervals
16513ff01b23SMartin Matuskawill reduce the import time but increase
16523ff01b23SMartin Matuskathe risk of failing to detect an active pool.
16533ff01b23SMartin MatuskaThe total activity check time is never allowed to drop below one second.
16543ff01b23SMartin Matuska.Pp
16553ff01b23SMartin MatuskaOn import the activity check waits a minimum amount of time determined by
1656e92ffd9bSMartin Matuska.Sy zfs_multihost_interval No \(mu Sy zfs_multihost_import_intervals ,
16573ff01b23SMartin Matuskaor the same product computed on the host which last had the pool imported,
16583ff01b23SMartin Matuskawhichever is greater.
16593ff01b23SMartin MatuskaThe activity check time may be further extended if the value of MMP
16603ff01b23SMartin Matuskadelay found in the best uberblock indicates actual multihost updates happened
16613ff01b23SMartin Matuskaat longer intervals than
16623ff01b23SMartin Matuska.Sy zfs_multihost_interval .
16633ff01b23SMartin MatuskaA minimum of
16643ff01b23SMartin Matuska.Em 100 ms
16653ff01b23SMartin Matuskais enforced.
16663ff01b23SMartin Matuska.Pp
16673ff01b23SMartin Matuska.Sy 0 No is equivalent to Sy 1 .
16683ff01b23SMartin Matuska.
16693ff01b23SMartin Matuska.It Sy zfs_multihost_fail_intervals Ns = Ns Sy 10 Pq uint
16703ff01b23SMartin MatuskaControls the behavior of the pool when multihost write failures or delays are
16713ff01b23SMartin Matuskadetected.
16723ff01b23SMartin Matuska.Pp
16733ff01b23SMartin MatuskaWhen
16743ff01b23SMartin Matuska.Sy 0 ,
16753ff01b23SMartin Matuskamultihost write failures or delays are ignored.
16763ff01b23SMartin MatuskaThe failures will still be reported to the ZED which depending on
16773ff01b23SMartin Matuskaits configuration may take action such as suspending the pool or offlining a
16783ff01b23SMartin Matuskadevice.
16793ff01b23SMartin Matuska.Pp
16803ff01b23SMartin MatuskaOtherwise, the pool will be suspended if
1681e92ffd9bSMartin Matuska.Sy zfs_multihost_fail_intervals No \(mu Sy zfs_multihost_interval
16823ff01b23SMartin Matuskamilliseconds pass without a successful MMP write.
16833ff01b23SMartin MatuskaThis guarantees the activity test will see MMP writes if the pool is imported.
16843ff01b23SMartin Matuska.Sy 1 No is equivalent to Sy 2 ;
16853ff01b23SMartin Matuskathis is necessary to prevent the pool from being suspended
16863ff01b23SMartin Matuskadue to normal, small I/O latency variations.
16873ff01b23SMartin Matuska.
16883ff01b23SMartin Matuska.It Sy zfs_no_scrub_io Ns = Ns Sy 0 Ns | Ns 1 Pq int
16893ff01b23SMartin MatuskaSet to disable scrub I/O.
16903ff01b23SMartin MatuskaThis results in scrubs not actually scrubbing data and
16913ff01b23SMartin Matuskasimply doing a metadata crawl of the pool instead.
16923ff01b23SMartin Matuska.
16933ff01b23SMartin Matuska.It Sy zfs_no_scrub_prefetch Ns = Ns Sy 0 Ns | Ns 1 Pq int
16943ff01b23SMartin MatuskaSet to disable block prefetching for scrubs.
16953ff01b23SMartin Matuska.
16963ff01b23SMartin Matuska.It Sy zfs_nocacheflush Ns = Ns Sy 0 Ns | Ns 1 Pq int
16973ff01b23SMartin MatuskaDisable cache flush operations on disks when writing.
16983ff01b23SMartin MatuskaSetting this will cause pool corruption on power loss
16993ff01b23SMartin Matuskaif a volatile out-of-order write cache is enabled.
17003ff01b23SMartin Matuska.
17013ff01b23SMartin Matuska.It Sy zfs_nopwrite_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
17023ff01b23SMartin MatuskaAllow no-operation writes.
17033ff01b23SMartin MatuskaThe occurrence of nopwrites will further depend on other pool properties
17043ff01b23SMartin Matuska.Pq i.a. the checksumming and compression algorithms .
17053ff01b23SMartin Matuska.
1706681ce946SMartin Matuska.It Sy zfs_dmu_offset_next_sync Ns = Ns Sy 1 Ns | Ns 0 Pq int
17073ff01b23SMartin MatuskaEnable forcing TXG sync to find holes.
1708681ce946SMartin MatuskaWhen enabled forces ZFS to sync data when
17093ff01b23SMartin Matuska.Sy SEEK_HOLE No or Sy SEEK_DATA
1710681ce946SMartin Matuskaflags are used allowing holes in a file to be accurately reported.
1711681ce946SMartin MatuskaWhen disabled holes will not be reported in recently dirtied files.
17123ff01b23SMartin Matuska.
1713716fd348SMartin Matuska.It Sy zfs_pd_bytes_max Ns = Ns Sy 52428800 Ns B Po 50 MiB Pc Pq int
17143ff01b23SMartin MatuskaThe number of bytes which should be prefetched during a pool traversal, like
17153ff01b23SMartin Matuska.Nm zfs Cm send
17163ff01b23SMartin Matuskaor other data crawling operations.
17173ff01b23SMartin Matuska.
1718be181ee2SMartin Matuska.It Sy zfs_traverse_indirect_prefetch_limit Ns = Ns Sy 32 Pq uint
17193ff01b23SMartin MatuskaThe number of blocks pointed by indirect (non-L0) block which should be
17203ff01b23SMartin Matuskaprefetched during a pool traversal, like
17213ff01b23SMartin Matuska.Nm zfs Cm send
17223ff01b23SMartin Matuskaor other data crawling operations.
17233ff01b23SMartin Matuska.
1724dbd5678dSMartin Matuska.It Sy zfs_per_txg_dirty_frees_percent Ns = Ns Sy 30 Ns % Pq u64
17253ff01b23SMartin MatuskaControl percentage of dirtied indirect blocks from frees allowed into one TXG.
17263ff01b23SMartin MatuskaAfter this threshold is crossed, additional frees will wait until the next TXG.
17273ff01b23SMartin Matuska.Sy 0 No disables this throttle .
17283ff01b23SMartin Matuska.
17293ff01b23SMartin Matuska.It Sy zfs_prefetch_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int
17303ff01b23SMartin MatuskaDisable predictive prefetch.
1731c03c5b1cSMartin MatuskaNote that it leaves "prescient" prefetch
1732c03c5b1cSMartin Matuska.Pq for, e.g., Nm zfs Cm send
17333ff01b23SMartin Matuskaintact.
17343ff01b23SMartin MatuskaUnlike predictive prefetch, prescient prefetch never issues I/O
17353ff01b23SMartin Matuskathat ends up not being needed, so it can't hurt performance.
17363ff01b23SMartin Matuska.
17373ff01b23SMartin Matuska.It Sy zfs_qat_checksum_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int
17383ff01b23SMartin MatuskaDisable QAT hardware acceleration for SHA256 checksums.
17393ff01b23SMartin MatuskaMay be unset after the ZFS modules have been loaded to initialize the QAT
17403ff01b23SMartin Matuskahardware as long as support is compiled in and the QAT driver is present.
17413ff01b23SMartin Matuska.
17423ff01b23SMartin Matuska.It Sy zfs_qat_compress_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int
17433ff01b23SMartin MatuskaDisable QAT hardware acceleration for gzip compression.
17443ff01b23SMartin MatuskaMay be unset after the ZFS modules have been loaded to initialize the QAT
17453ff01b23SMartin Matuskahardware as long as support is compiled in and the QAT driver is present.
17463ff01b23SMartin Matuska.
17473ff01b23SMartin Matuska.It Sy zfs_qat_encrypt_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int
17483ff01b23SMartin MatuskaDisable QAT hardware acceleration for AES-GCM encryption.
17493ff01b23SMartin MatuskaMay be unset after the ZFS modules have been loaded to initialize the QAT
17503ff01b23SMartin Matuskahardware as long as support is compiled in and the QAT driver is present.
17513ff01b23SMartin Matuska.
1752dbd5678dSMartin Matuska.It Sy zfs_vnops_read_chunk_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64
17533ff01b23SMartin MatuskaBytes to read per chunk.
17543ff01b23SMartin Matuska.
1755be181ee2SMartin Matuska.It Sy zfs_read_history Ns = Ns Sy 0 Pq uint
17563ff01b23SMartin MatuskaHistorical statistics for this many latest reads will be available in
17573ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /reads .
17583ff01b23SMartin Matuska.
17593ff01b23SMartin Matuska.It Sy zfs_read_history_hits Ns = Ns Sy 0 Ns | Ns 1 Pq int
17603ff01b23SMartin MatuskaInclude cache hits in read history
17613ff01b23SMartin Matuska.
1762dbd5678dSMartin Matuska.It Sy zfs_rebuild_max_segment Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64
17633ff01b23SMartin MatuskaMaximum read segment size to issue when sequentially resilvering a
17643ff01b23SMartin Matuskatop-level vdev.
17653ff01b23SMartin Matuska.
17663ff01b23SMartin Matuska.It Sy zfs_rebuild_scrub_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
17673ff01b23SMartin MatuskaAutomatically start a pool scrub when the last active sequential resilver
17683ff01b23SMartin Matuskacompletes in order to verify the checksums of all blocks which have been
17693ff01b23SMartin Matuskaresilvered.
17703ff01b23SMartin MatuskaThis is enabled by default and strongly recommended.
17713ff01b23SMartin Matuska.
1772dbd5678dSMartin Matuska.It Sy zfs_rebuild_vdev_limit Ns = Ns Sy 33554432 Ns B Po 32 MiB Pc Pq u64
17733ff01b23SMartin MatuskaMaximum amount of I/O that can be concurrently issued for a sequential
17743ff01b23SMartin Matuskaresilver per leaf device, given in bytes.
17753ff01b23SMartin Matuska.
17763ff01b23SMartin Matuska.It Sy zfs_reconstruct_indirect_combinations_max Ns = Ns Sy 4096 Pq int
17773ff01b23SMartin MatuskaIf an indirect split block contains more than this many possible unique
17783ff01b23SMartin Matuskacombinations when being reconstructed, consider it too computationally
17793ff01b23SMartin Matuskaexpensive to check them all.
17803ff01b23SMartin MatuskaInstead, try at most this many randomly selected
17813ff01b23SMartin Matuskacombinations each time the block is accessed.
17823ff01b23SMartin MatuskaThis allows all segment copies to participate fairly
17833ff01b23SMartin Matuskain the reconstruction when all combinations
17843ff01b23SMartin Matuskacannot be checked and prevents repeated use of one bad copy.
17853ff01b23SMartin Matuska.
17863ff01b23SMartin Matuska.It Sy zfs_recover Ns = Ns Sy 0 Ns | Ns 1 Pq int
17873ff01b23SMartin MatuskaSet to attempt to recover from fatal errors.
17883ff01b23SMartin MatuskaThis should only be used as a last resort,
17893ff01b23SMartin Matuskaas it typically results in leaked space, or worse.
17903ff01b23SMartin Matuska.
17913ff01b23SMartin Matuska.It Sy zfs_removal_ignore_errors Ns = Ns Sy 0 Ns | Ns 1 Pq int
1792c03c5b1cSMartin MatuskaIgnore hard I/O errors during device removal.
1793c03c5b1cSMartin MatuskaWhen set, if a device encounters a hard I/O error during the removal process
17943ff01b23SMartin Matuskathe removal will not be cancelled.
17953ff01b23SMartin MatuskaThis can result in a normally recoverable block becoming permanently damaged
17963ff01b23SMartin Matuskaand is hence not recommended.
17973ff01b23SMartin MatuskaThis should only be used as a last resort when the
17983ff01b23SMartin Matuskapool cannot be returned to a healthy state prior to removing the device.
17993ff01b23SMartin Matuska.
1800be181ee2SMartin Matuska.It Sy zfs_removal_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq uint
18013ff01b23SMartin MatuskaThis is used by the test suite so that it can ensure that certain actions
18023ff01b23SMartin Matuskahappen while in the middle of a removal.
18033ff01b23SMartin Matuska.
1804be181ee2SMartin Matuska.It Sy zfs_remove_max_segment Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint
18053ff01b23SMartin MatuskaThe largest contiguous segment that we will attempt to allocate when removing
18063ff01b23SMartin Matuskaa device.
18073ff01b23SMartin MatuskaIf there is a performance problem with attempting to allocate large blocks,
18083ff01b23SMartin Matuskaconsider decreasing this.
18093ff01b23SMartin MatuskaThe default value is also the maximum.
18103ff01b23SMartin Matuska.
18113ff01b23SMartin Matuska.It Sy zfs_resilver_disable_defer Ns = Ns Sy 0 Ns | Ns 1 Pq int
18123ff01b23SMartin MatuskaIgnore the
18133ff01b23SMartin Matuska.Sy resilver_defer
18143ff01b23SMartin Matuskafeature, causing an operation that would start a resilver to
18153ff01b23SMartin Matuskaimmediately restart the one in progress.
18163ff01b23SMartin Matuska.
1817be181ee2SMartin Matuska.It Sy zfs_resilver_min_time_ms Ns = Ns Sy 3000 Ns ms Po 3 s Pc Pq uint
18183ff01b23SMartin MatuskaResilvers are processed by the sync thread.
18193ff01b23SMartin MatuskaWhile resilvering, it will spend at least this much time
18203ff01b23SMartin Matuskaworking on a resilver between TXG flushes.
18213ff01b23SMartin Matuska.
18223ff01b23SMartin Matuska.It Sy zfs_scan_ignore_errors Ns = Ns Sy 0 Ns | Ns 1 Pq int
18233ff01b23SMartin MatuskaIf set, remove the DTL (dirty time list) upon completion of a pool scan (scrub),
18243ff01b23SMartin Matuskaeven if there were unrepairable errors.
18253ff01b23SMartin MatuskaIntended to be used during pool repair or recovery to
18263ff01b23SMartin Matuskastop resilvering when the pool is next imported.
18273ff01b23SMartin Matuska.
1828be181ee2SMartin Matuska.It Sy zfs_scrub_min_time_ms Ns = Ns Sy 1000 Ns ms Po 1 s Pc Pq uint
18293ff01b23SMartin MatuskaScrubs are processed by the sync thread.
18303ff01b23SMartin MatuskaWhile scrubbing, it will spend at least this much time
18313ff01b23SMartin Matuskaworking on a scrub between TXG flushes.
18323ff01b23SMartin Matuska.
1833be181ee2SMartin Matuska.It Sy zfs_scan_checkpoint_intval Ns = Ns Sy 7200 Ns s Po 2 hour Pc Pq uint
18343ff01b23SMartin MatuskaTo preserve progress across reboots, the sequential scan algorithm periodically
18353ff01b23SMartin Matuskaneeds to stop metadata scanning and issue all the verification I/O to disk.
18363ff01b23SMartin MatuskaThe frequency of this flushing is determined by this tunable.
18373ff01b23SMartin Matuska.
1838be181ee2SMartin Matuska.It Sy zfs_scan_fill_weight Ns = Ns Sy 3 Pq uint
18393ff01b23SMartin MatuskaThis tunable affects how scrub and resilver I/O segments are ordered.
18403ff01b23SMartin MatuskaA higher number indicates that we care more about how filled in a segment is,
18413ff01b23SMartin Matuskawhile a lower number indicates we care more about the size of the extent without
18423ff01b23SMartin Matuskaconsidering the gaps within a segment.
18433ff01b23SMartin MatuskaThis value is only tunable upon module insertion.
1844bb2d13b6SMartin MatuskaChanging the value afterwards will have no effect on scrub or resilver
1845bb2d13b6SMartin Matuskaperformance.
18463ff01b23SMartin Matuska.
1847be181ee2SMartin Matuska.It Sy zfs_scan_issue_strategy Ns = Ns Sy 0 Pq uint
18483ff01b23SMartin MatuskaDetermines the order that data will be verified while scrubbing or resilvering:
18493ff01b23SMartin Matuska.Bl -tag -compact -offset 4n -width "a"
18503ff01b23SMartin Matuska.It Sy 1
18513ff01b23SMartin MatuskaData will be verified as sequentially as possible, given the
18523ff01b23SMartin Matuskaamount of memory reserved for scrubbing
18533ff01b23SMartin Matuska.Pq see Sy zfs_scan_mem_lim_fact .
18543ff01b23SMartin MatuskaThis may improve scrub performance if the pool's data is very fragmented.
18553ff01b23SMartin Matuska.It Sy 2
18563ff01b23SMartin MatuskaThe largest mostly-contiguous chunk of found data will be verified first.
18573ff01b23SMartin MatuskaBy deferring scrubbing of small segments, we may later find adjacent data
18583ff01b23SMartin Matuskato coalesce and increase the segment size.
18593ff01b23SMartin Matuska.It Sy 0
18603ff01b23SMartin Matuska.No Use strategy Sy 1 No during normal verification
18613ff01b23SMartin Matuska.No and strategy Sy 2 No while taking a checkpoint .
18623ff01b23SMartin Matuska.El
18633ff01b23SMartin Matuska.
18643ff01b23SMartin Matuska.It Sy zfs_scan_legacy Ns = Ns Sy 0 Ns | Ns 1 Pq int
18653ff01b23SMartin MatuskaIf unset, indicates that scrubs and resilvers will gather metadata in
18663ff01b23SMartin Matuskamemory before issuing sequential I/O.
18673ff01b23SMartin MatuskaOtherwise indicates that the legacy algorithm will be used,
18683ff01b23SMartin Matuskawhere I/O is initiated as soon as it is discovered.
18693ff01b23SMartin MatuskaUnsetting will not affect scrubs or resilvers that are already in progress.
18703ff01b23SMartin Matuska.
1871716fd348SMartin Matuska.It Sy zfs_scan_max_ext_gap Ns = Ns Sy 2097152 Ns B Po 2 MiB Pc Pq int
18723ff01b23SMartin MatuskaSets the largest gap in bytes between scrub/resilver I/O operations
18733ff01b23SMartin Matuskathat will still be considered sequential for sorting purposes.
18743ff01b23SMartin MatuskaChanging this value will not
18753ff01b23SMartin Matuskaaffect scrubs or resilvers that are already in progress.
18763ff01b23SMartin Matuska.
1877be181ee2SMartin Matuska.It Sy zfs_scan_mem_lim_fact Ns = Ns Sy 20 Ns ^-1 Pq uint
18783ff01b23SMartin MatuskaMaximum fraction of RAM used for I/O sorting by sequential scan algorithm.
18793ff01b23SMartin MatuskaThis tunable determines the hard limit for I/O sorting memory usage.
18803ff01b23SMartin MatuskaWhen the hard limit is reached we stop scanning metadata and start issuing
18813ff01b23SMartin Matuskadata verification I/O.
18823ff01b23SMartin MatuskaThis is done until we get below the soft limit.
18833ff01b23SMartin Matuska.
1884be181ee2SMartin Matuska.It Sy zfs_scan_mem_lim_soft_fact Ns = Ns Sy 20 Ns ^-1 Pq uint
18853ff01b23SMartin MatuskaThe fraction of the hard limit used to determined the soft limit for I/O sorting
18863ff01b23SMartin Matuskaby the sequential scan algorithm.
18873ff01b23SMartin MatuskaWhen we cross this limit from below no action is taken.
1888bb2d13b6SMartin MatuskaWhen we cross this limit from above it is because we are issuing verification
1889bb2d13b6SMartin MatuskaI/O.
18903ff01b23SMartin MatuskaIn this case (unless the metadata scan is done) we stop issuing verification I/O
18913ff01b23SMartin Matuskaand start scanning metadata again until we get to the hard limit.
18923ff01b23SMartin Matuska.
18933ff01b23SMartin Matuska.It Sy zfs_scan_strict_mem_lim Ns = Ns Sy 0 Ns | Ns 1 Pq int
18943ff01b23SMartin MatuskaEnforce tight memory limits on pool scans when a sequential scan is in progress.
18953ff01b23SMartin MatuskaWhen disabled, the memory limit may be exceeded by fast disks.
18963ff01b23SMartin Matuska.
18973ff01b23SMartin Matuska.It Sy zfs_scan_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq int
18983ff01b23SMartin MatuskaFreezes a scrub/resilver in progress without actually pausing it.
18993ff01b23SMartin MatuskaIntended for testing/debugging.
19003ff01b23SMartin Matuska.
1901716fd348SMartin Matuska.It Sy zfs_scan_vdev_limit Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq int
19023ff01b23SMartin MatuskaMaximum amount of data that can be concurrently issued at once for scrubs and
19033ff01b23SMartin Matuskaresilvers per leaf device, given in bytes.
19043ff01b23SMartin Matuska.
19053ff01b23SMartin Matuska.It Sy zfs_send_corrupt_data Ns = Ns Sy 0 Ns | Ns 1 Pq int
19063ff01b23SMartin MatuskaAllow sending of corrupt data (ignore read/checksum errors when sending).
19073ff01b23SMartin Matuska.
19083ff01b23SMartin Matuska.It Sy zfs_send_unmodified_spill_blocks Ns = Ns Sy 1 Ns | Ns 0 Pq int
19093ff01b23SMartin MatuskaInclude unmodified spill blocks in the send stream.
19103ff01b23SMartin MatuskaUnder certain circumstances, previous versions of ZFS could incorrectly
19113ff01b23SMartin Matuskaremove the spill block from an existing object.
19123ff01b23SMartin MatuskaIncluding unmodified copies of the spill blocks creates a backwards-compatible
19133ff01b23SMartin Matuskastream which will recreate a spill block if it was incorrectly removed.
19143ff01b23SMartin Matuska.
1915be181ee2SMartin Matuska.It Sy zfs_send_no_prefetch_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq uint
19163ff01b23SMartin MatuskaThe fill fraction of the
19173ff01b23SMartin Matuska.Nm zfs Cm send
19183ff01b23SMartin Matuskainternal queues.
19193ff01b23SMartin MatuskaThe fill fraction controls the timing with which internal threads are woken up.
19203ff01b23SMartin Matuska.
1921be181ee2SMartin Matuska.It Sy zfs_send_no_prefetch_queue_length Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq uint
19223ff01b23SMartin MatuskaThe maximum number of bytes allowed in
19233ff01b23SMartin Matuska.Nm zfs Cm send Ns 's
19243ff01b23SMartin Matuskainternal queues.
19253ff01b23SMartin Matuska.
1926be181ee2SMartin Matuska.It Sy zfs_send_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq uint
19273ff01b23SMartin MatuskaThe fill fraction of the
19283ff01b23SMartin Matuska.Nm zfs Cm send
19293ff01b23SMartin Matuskaprefetch queue.
19303ff01b23SMartin MatuskaThe fill fraction controls the timing with which internal threads are woken up.
19313ff01b23SMartin Matuska.
1932be181ee2SMartin Matuska.It Sy zfs_send_queue_length Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint
19333ff01b23SMartin MatuskaThe maximum number of bytes allowed that will be prefetched by
19343ff01b23SMartin Matuska.Nm zfs Cm send .
19353ff01b23SMartin MatuskaThis value must be at least twice the maximum block size in use.
19363ff01b23SMartin Matuska.
1937be181ee2SMartin Matuska.It Sy zfs_recv_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq uint
19383ff01b23SMartin MatuskaThe fill fraction of the
19393ff01b23SMartin Matuska.Nm zfs Cm receive
19403ff01b23SMartin Matuskaqueue.
19413ff01b23SMartin MatuskaThe fill fraction controls the timing with which internal threads are woken up.
19423ff01b23SMartin Matuska.
1943be181ee2SMartin Matuska.It Sy zfs_recv_queue_length Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint
19443ff01b23SMartin MatuskaThe maximum number of bytes allowed in the
19453ff01b23SMartin Matuska.Nm zfs Cm receive
19463ff01b23SMartin Matuskaqueue.
19473ff01b23SMartin MatuskaThis value must be at least twice the maximum block size in use.
19483ff01b23SMartin Matuska.
1949be181ee2SMartin Matuska.It Sy zfs_recv_write_batch_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq uint
19503ff01b23SMartin MatuskaThe maximum amount of data, in bytes, that
19513ff01b23SMartin Matuska.Nm zfs Cm receive
19523ff01b23SMartin Matuskawill write in one DMU transaction.
19533ff01b23SMartin MatuskaThis is the uncompressed size, even when receiving a compressed send stream.
19543ff01b23SMartin MatuskaThis setting will not reduce the write size below a single block.
19553ff01b23SMartin MatuskaCapped at a maximum of
1956716fd348SMartin Matuska.Sy 32 MiB .
19573ff01b23SMartin Matuska.
1958271171e0SMartin Matuska.It Sy zfs_recv_best_effort_corrective Ns = Ns Sy 0 Pq int
1959271171e0SMartin MatuskaWhen this variable is set to non-zero a corrective receive:
1960271171e0SMartin Matuska.Bl -enum -compact -offset 4n -width "1."
1961271171e0SMartin Matuska.It
1962271171e0SMartin MatuskaDoes not enforce the restriction of source & destination snapshot GUIDs
1963271171e0SMartin Matuskamatching.
1964271171e0SMartin Matuska.It
1965271171e0SMartin MatuskaIf there is an error during healing, the healing receive is not
1966271171e0SMartin Matuskaterminated instead it moves on to the next record.
1967271171e0SMartin Matuska.El
1968271171e0SMartin Matuska.
1969be181ee2SMartin Matuska.It Sy zfs_override_estimate_recordsize Ns = Ns Sy 0 Ns | Ns 1 Pq uint
19703ff01b23SMartin MatuskaSetting this variable overrides the default logic for estimating block
19713ff01b23SMartin Matuskasizes when doing a
19723ff01b23SMartin Matuska.Nm zfs Cm send .
19733ff01b23SMartin MatuskaThe default heuristic is that the average block size
19743ff01b23SMartin Matuskawill be the current recordsize.
19753ff01b23SMartin MatuskaOverride this value if most data in your dataset is not of that size
19763ff01b23SMartin Matuskaand you require accurate zfs send size estimates.
19773ff01b23SMartin Matuska.
1978be181ee2SMartin Matuska.It Sy zfs_sync_pass_deferred_free Ns = Ns Sy 2 Pq uint
19793ff01b23SMartin MatuskaFlushing of data to disk is done in passes.
19803ff01b23SMartin MatuskaDefer frees starting in this pass.
19813ff01b23SMartin Matuska.
1982716fd348SMartin Matuska.It Sy zfs_spa_discard_memory_limit Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq int
19833ff01b23SMartin MatuskaMaximum memory used for prefetching a checkpoint's space map on each
19843ff01b23SMartin Matuskavdev while discarding the checkpoint.
19853ff01b23SMartin Matuska.
1986be181ee2SMartin Matuska.It Sy zfs_special_class_metadata_reserve_pct Ns = Ns Sy 25 Ns % Pq uint
19873ff01b23SMartin MatuskaOnly allow small data blocks to be allocated on the special and dedup vdev
1988bb2d13b6SMartin Matuskatypes when the available free space percentage on these vdevs exceeds this
1989bb2d13b6SMartin Matuskavalue.
19903ff01b23SMartin MatuskaThis ensures reserved space is available for pool metadata as the
19913ff01b23SMartin Matuskaspecial vdevs approach capacity.
19923ff01b23SMartin Matuska.
1993be181ee2SMartin Matuska.It Sy zfs_sync_pass_dont_compress Ns = Ns Sy 8 Pq uint
19943ff01b23SMartin MatuskaStarting in this sync pass, disable compression (including of metadata).
19953ff01b23SMartin MatuskaWith the default setting, in practice, we don't have this many sync passes,
19963ff01b23SMartin Matuskaso this has no effect.
19973ff01b23SMartin Matuska.Pp
19983ff01b23SMartin MatuskaThe original intent was that disabling compression would help the sync passes
19993ff01b23SMartin Matuskato converge.
20003ff01b23SMartin MatuskaHowever, in practice, disabling compression increases
20013ff01b23SMartin Matuskathe average number of sync passes; because when we turn compression off,
20023ff01b23SMartin Matuskamany blocks' size will change, and thus we have to re-allocate
20033ff01b23SMartin Matuska(not overwrite) them.
20043ff01b23SMartin MatuskaIt also increases the number of
2005716fd348SMartin Matuska.Em 128 KiB
20063ff01b23SMartin Matuskaallocations (e.g. for indirect blocks and spacemaps)
20073ff01b23SMartin Matuskabecause these will not be compressed.
20083ff01b23SMartin MatuskaThe
2009716fd348SMartin Matuska.Em 128 KiB
20103ff01b23SMartin Matuskaallocations are especially detrimental to performance
2011bb2d13b6SMartin Matuskaon highly fragmented systems, which may have very few free segments of this
2012bb2d13b6SMartin Matuskasize,
20133ff01b23SMartin Matuskaand may need to load new metaslabs to satisfy these allocations.
20143ff01b23SMartin Matuska.
2015be181ee2SMartin Matuska.It Sy zfs_sync_pass_rewrite Ns = Ns Sy 2 Pq uint
20163ff01b23SMartin MatuskaRewrite new block pointers starting in this pass.
20173ff01b23SMartin Matuska.
20183ff01b23SMartin Matuska.It Sy zfs_sync_taskq_batch_pct Ns = Ns Sy 75 Ns % Pq int
20193ff01b23SMartin MatuskaThis controls the number of threads used by
20203ff01b23SMartin Matuska.Sy dp_sync_taskq .
20213ff01b23SMartin MatuskaThe default value of
20223ff01b23SMartin Matuska.Sy 75%
20233ff01b23SMartin Matuskawill create a maximum of one thread per CPU.
20243ff01b23SMartin Matuska.
2025716fd348SMartin Matuska.It Sy zfs_trim_extent_bytes_max Ns = Ns Sy 134217728 Ns B Po 128 MiB Pc Pq uint
20263ff01b23SMartin MatuskaMaximum size of TRIM command.
2027bb2d13b6SMartin MatuskaLarger ranges will be split into chunks no larger than this value before
2028bb2d13b6SMartin Matuskaissuing.
20293ff01b23SMartin Matuska.
2030716fd348SMartin Matuska.It Sy zfs_trim_extent_bytes_min Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint
20313ff01b23SMartin MatuskaMinimum size of TRIM commands.
20323ff01b23SMartin MatuskaTRIM ranges smaller than this will be skipped,
20333ff01b23SMartin Matuskaunless they're part of a larger range which was chunked.
20343ff01b23SMartin MatuskaThis is done because it's common for these small TRIMs
20353ff01b23SMartin Matuskato negatively impact overall performance.
20363ff01b23SMartin Matuska.
20373ff01b23SMartin Matuska.It Sy zfs_trim_metaslab_skip Ns = Ns Sy 0 Ns | Ns 1 Pq uint
20383ff01b23SMartin MatuskaSkip uninitialized metaslabs during the TRIM process.
2039bb2d13b6SMartin MatuskaThis option is useful for pools constructed from large thinly-provisioned
2040bb2d13b6SMartin Matuskadevices
20413ff01b23SMartin Matuskawhere TRIM operations are slow.
20423ff01b23SMartin MatuskaAs a pool ages, an increasing fraction of the pool's metaslabs
20433ff01b23SMartin Matuskawill be initialized, progressively degrading the usefulness of this option.
20443ff01b23SMartin MatuskaThis setting is stored when starting a manual TRIM and will
20453ff01b23SMartin Matuskapersist for the duration of the requested TRIM.
20463ff01b23SMartin Matuska.
20473ff01b23SMartin Matuska.It Sy zfs_trim_queue_limit Ns = Ns Sy 10 Pq uint
20483ff01b23SMartin MatuskaMaximum number of queued TRIMs outstanding per leaf vdev.
20493ff01b23SMartin MatuskaThe number of concurrent TRIM commands issued to the device is controlled by
20503ff01b23SMartin Matuska.Sy zfs_vdev_trim_min_active No and Sy zfs_vdev_trim_max_active .
20513ff01b23SMartin Matuska.
20523ff01b23SMartin Matuska.It Sy zfs_trim_txg_batch Ns = Ns Sy 32 Pq uint
20533ff01b23SMartin MatuskaThe number of transaction groups' worth of frees which should be aggregated
20543ff01b23SMartin Matuskabefore TRIM operations are issued to the device.
20553ff01b23SMartin MatuskaThis setting represents a trade-off between issuing larger,
20563ff01b23SMartin Matuskamore efficient TRIM operations and the delay
20573ff01b23SMartin Matuskabefore the recently trimmed space is available for use by the device.
20583ff01b23SMartin Matuska.Pp
20593ff01b23SMartin MatuskaIncreasing this value will allow frees to be aggregated for a longer time.
2060bb2d13b6SMartin MatuskaThis will result is larger TRIM operations and potentially increased memory
2061bb2d13b6SMartin Matuskausage.
20623ff01b23SMartin MatuskaDecreasing this value will have the opposite effect.
20633ff01b23SMartin MatuskaThe default of
20643ff01b23SMartin Matuska.Sy 32
20653ff01b23SMartin Matuskawas determined to be a reasonable compromise.
20663ff01b23SMartin Matuska.
2067be181ee2SMartin Matuska.It Sy zfs_txg_history Ns = Ns Sy 0 Pq uint
20683ff01b23SMartin MatuskaHistorical statistics for this many latest TXGs will be available in
20693ff01b23SMartin Matuska.Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /TXGs .
20703ff01b23SMartin Matuska.
2071be181ee2SMartin Matuska.It Sy zfs_txg_timeout Ns = Ns Sy 5 Ns s Pq uint
2072bb2d13b6SMartin MatuskaFlush dirty data to disk at least every this many seconds (maximum TXG
2073bb2d13b6SMartin Matuskaduration).
20743ff01b23SMartin Matuska.
2075be181ee2SMartin Matuska.It Sy zfs_vdev_aggregate_trim Ns = Ns Sy 0 Ns | Ns 1 Pq uint
2076c03c5b1cSMartin MatuskaAllow TRIM I/O operations to be aggregated.
20773ff01b23SMartin MatuskaThis is normally not helpful because the extents to be trimmed
20783ff01b23SMartin Matuskawill have been already been aggregated by the metaslab.
20793ff01b23SMartin MatuskaThis option is provided for debugging and performance analysis.
20803ff01b23SMartin Matuska.
2081be181ee2SMartin Matuska.It Sy zfs_vdev_aggregation_limit Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq uint
20823ff01b23SMartin MatuskaMax vdev I/O aggregation size.
20833ff01b23SMartin Matuska.
2084be181ee2SMartin Matuska.It Sy zfs_vdev_aggregation_limit_non_rotating Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint
20853ff01b23SMartin MatuskaMax vdev I/O aggregation size for non-rotating media.
20863ff01b23SMartin Matuska.
2087be181ee2SMartin Matuska.It Sy zfs_vdev_cache_bshift Ns = Ns Sy 16 Po 64 KiB Pc Pq uint
20883ff01b23SMartin MatuskaShift size to inflate reads to.
20893ff01b23SMartin Matuska.
2090be181ee2SMartin Matuska.It Sy zfs_vdev_cache_max Ns = Ns Sy 16384 Ns B Po 16 KiB Pc Pq uint
20913ff01b23SMartin MatuskaInflate reads smaller than this value to meet the
20923ff01b23SMartin Matuska.Sy zfs_vdev_cache_bshift
20933ff01b23SMartin Matuskasize
2094716fd348SMartin Matuska.Pq default Sy 64 KiB .
20953ff01b23SMartin Matuska.
2096be181ee2SMartin Matuska.It Sy zfs_vdev_cache_size Ns = Ns Sy 0 Pq uint
20973ff01b23SMartin MatuskaTotal size of the per-disk cache in bytes.
20983ff01b23SMartin Matuska.Pp
20993ff01b23SMartin MatuskaCurrently this feature is disabled, as it has been found to not be helpful
21003ff01b23SMartin Matuskafor performance and in some cases harmful.
21013ff01b23SMartin Matuska.
21023ff01b23SMartin Matuska.It Sy zfs_vdev_mirror_rotating_inc Ns = Ns Sy 0 Pq int
21033ff01b23SMartin MatuskaA number by which the balancing algorithm increments the load calculation for
21043ff01b23SMartin Matuskathe purpose of selecting the least busy mirror member when an I/O operation
21053ff01b23SMartin Matuskaimmediately follows its predecessor on rotational vdevs
21063ff01b23SMartin Matuskafor the purpose of making decisions based on load.
21073ff01b23SMartin Matuska.
21083ff01b23SMartin Matuska.It Sy zfs_vdev_mirror_rotating_seek_inc Ns = Ns Sy 5 Pq int
21093ff01b23SMartin MatuskaA number by which the balancing algorithm increments the load calculation for
21103ff01b23SMartin Matuskathe purpose of selecting the least busy mirror member when an I/O operation
21113ff01b23SMartin Matuskalacks locality as defined by
21123ff01b23SMartin Matuska.Sy zfs_vdev_mirror_rotating_seek_offset .
21133ff01b23SMartin MatuskaOperations within this that are not immediately following the previous operation
21143ff01b23SMartin Matuskaare incremented by half.
21153ff01b23SMartin Matuska.
2116716fd348SMartin Matuska.It Sy zfs_vdev_mirror_rotating_seek_offset Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq int
21173ff01b23SMartin MatuskaThe maximum distance for the last queued I/O operation in which
21183ff01b23SMartin Matuskathe balancing algorithm considers an operation to have locality.
21193ff01b23SMartin Matuska.No See Sx ZFS I/O SCHEDULER .
21203ff01b23SMartin Matuska.
21213ff01b23SMartin Matuska.It Sy zfs_vdev_mirror_non_rotating_inc Ns = Ns Sy 0 Pq int
21223ff01b23SMartin MatuskaA number by which the balancing algorithm increments the load calculation for
21233ff01b23SMartin Matuskathe purpose of selecting the least busy mirror member on non-rotational vdevs
21243ff01b23SMartin Matuskawhen I/O operations do not immediately follow one another.
21253ff01b23SMartin Matuska.
21263ff01b23SMartin Matuska.It Sy zfs_vdev_mirror_non_rotating_seek_inc Ns = Ns Sy 1 Pq int
21273ff01b23SMartin MatuskaA number by which the balancing algorithm increments the load calculation for
2128bb2d13b6SMartin Matuskathe purpose of selecting the least busy mirror member when an I/O operation
2129bb2d13b6SMartin Matuskalacks
21303ff01b23SMartin Matuskalocality as defined by the
21313ff01b23SMartin Matuska.Sy zfs_vdev_mirror_rotating_seek_offset .
21323ff01b23SMartin MatuskaOperations within this that are not immediately following the previous operation
21333ff01b23SMartin Matuskaare incremented by half.
21343ff01b23SMartin Matuska.
2135be181ee2SMartin Matuska.It Sy zfs_vdev_read_gap_limit Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint
21363ff01b23SMartin MatuskaAggregate read I/O operations if the on-disk gap between them is within this
21373ff01b23SMartin Matuskathreshold.
21383ff01b23SMartin Matuska.
2139be181ee2SMartin Matuska.It Sy zfs_vdev_write_gap_limit Ns = Ns Sy 4096 Ns B Po 4 KiB Pc Pq uint
21403ff01b23SMartin MatuskaAggregate write I/O operations if the on-disk gap between them is within this
21413ff01b23SMartin Matuskathreshold.
21423ff01b23SMartin Matuska.
21433ff01b23SMartin Matuska.It Sy zfs_vdev_raidz_impl Ns = Ns Sy fastest Pq string
21443ff01b23SMartin MatuskaSelect the raidz parity implementation to use.
21453ff01b23SMartin Matuska.Pp
21463ff01b23SMartin MatuskaVariants that don't depend on CPU-specific features
21473ff01b23SMartin Matuskamay be selected on module load, as they are supported on all systems.
21483ff01b23SMartin MatuskaThe remaining options may only be set after the module is loaded,
21493ff01b23SMartin Matuskaas they are available only if the implementations are compiled in
21503ff01b23SMartin Matuskaand supported on the running system.
21513ff01b23SMartin Matuska.Pp
21523ff01b23SMartin MatuskaOnce the module is loaded,
21533ff01b23SMartin Matuska.Pa /sys/module/zfs/parameters/zfs_vdev_raidz_impl
21543ff01b23SMartin Matuskawill show the available options,
21553ff01b23SMartin Matuskawith the currently selected one enclosed in square brackets.
21563ff01b23SMartin Matuska.Pp
21573ff01b23SMartin Matuska.TS
21583ff01b23SMartin Matuskalb l l .
21593ff01b23SMartin Matuskafastest	selected by built-in benchmark
21603ff01b23SMartin Matuskaoriginal	original implementation
21613ff01b23SMartin Matuskascalar	scalar implementation
21623ff01b23SMartin Matuskasse2	SSE2 instruction set	64-bit x86
21633ff01b23SMartin Matuskassse3	SSSE3 instruction set	64-bit x86
21643ff01b23SMartin Matuskaavx2	AVX2 instruction set	64-bit x86
21653ff01b23SMartin Matuskaavx512f	AVX512F instruction set	64-bit x86
21663ff01b23SMartin Matuskaavx512bw	AVX512F & AVX512BW instruction sets	64-bit x86
21673ff01b23SMartin Matuskaaarch64_neon	NEON	Aarch64/64-bit ARMv8
21683ff01b23SMartin Matuskaaarch64_neonx2	NEON with more unrolling	Aarch64/64-bit ARMv8
21693ff01b23SMartin Matuskapowerpc_altivec	Altivec	PowerPC
21703ff01b23SMartin Matuska.TE
21713ff01b23SMartin Matuska.
21723ff01b23SMartin Matuska.It Sy zfs_vdev_scheduler Pq charp
21733ff01b23SMartin Matuska.Sy DEPRECATED .
21742faf504dSMartin MatuskaPrints warning to kernel log for compatibility.
21753ff01b23SMartin Matuska.
2176be181ee2SMartin Matuska.It Sy zfs_zevent_len_max Ns = Ns Sy 512 Pq uint
21773ff01b23SMartin MatuskaMax event queue length.
21783ff01b23SMartin MatuskaEvents in the queue can be viewed with
21793ff01b23SMartin Matuska.Xr zpool-events 8 .
21803ff01b23SMartin Matuska.
21813ff01b23SMartin Matuska.It Sy zfs_zevent_retain_max Ns = Ns Sy 2000 Pq int
21823ff01b23SMartin MatuskaMaximum recent zevent records to retain for duplicate checking.
21833ff01b23SMartin MatuskaSetting this to
21843ff01b23SMartin Matuska.Sy 0
21853ff01b23SMartin Matuskadisables duplicate detection.
21863ff01b23SMartin Matuska.
21873ff01b23SMartin Matuska.It Sy zfs_zevent_retain_expire_secs Ns = Ns Sy 900 Ns s Po 15 min Pc Pq int
21883ff01b23SMartin MatuskaLifespan for a recent ereport that was retained for duplicate checking.
21893ff01b23SMartin Matuska.
21903ff01b23SMartin Matuska.It Sy zfs_zil_clean_taskq_maxalloc Ns = Ns Sy 1048576 Pq int
21913ff01b23SMartin MatuskaThe maximum number of taskq entries that are allowed to be cached.
21923ff01b23SMartin MatuskaWhen this limit is exceeded transaction records (itxs)
21933ff01b23SMartin Matuskawill be cleaned synchronously.
21943ff01b23SMartin Matuska.
21953ff01b23SMartin Matuska.It Sy zfs_zil_clean_taskq_minalloc Ns = Ns Sy 1024 Pq int
21963ff01b23SMartin MatuskaThe number of taskq entries that are pre-populated when the taskq is first
21973ff01b23SMartin Matuskacreated and are immediately available for use.
21983ff01b23SMartin Matuska.
21993ff01b23SMartin Matuska.It Sy zfs_zil_clean_taskq_nthr_pct Ns = Ns Sy 100 Ns % Pq int
22003ff01b23SMartin MatuskaThis controls the number of threads used by
22013ff01b23SMartin Matuska.Sy dp_zil_clean_taskq .
22023ff01b23SMartin MatuskaThe default value of
22033ff01b23SMartin Matuska.Sy 100%
22043ff01b23SMartin Matuskawill create a maximum of one thread per cpu.
22053ff01b23SMartin Matuska.
2206be181ee2SMartin Matuska.It Sy zil_maxblocksize Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint
22073ff01b23SMartin MatuskaThis sets the maximum block size used by the ZIL.
22083ff01b23SMartin MatuskaOn very fragmented pools, lowering this
2209716fd348SMartin Matuska.Pq typically to Sy 36 KiB
22103ff01b23SMartin Matuskacan improve performance.
22113ff01b23SMartin Matuska.
2212*15f0b8c3SMartin Matuska.It Sy zil_min_commit_timeout Ns = Ns Sy 5000 Pq u64
2213*15f0b8c3SMartin MatuskaThis sets the minimum delay in nanoseconds ZIL care to delay block commit,
2214*15f0b8c3SMartin Matuskawaiting for more records.
2215*15f0b8c3SMartin MatuskaIf ZIL writes are too fast, kernel may not be able sleep for so short interval,
2216*15f0b8c3SMartin Matuskaincreasing log latency above allowed by
2217*15f0b8c3SMartin Matuska.Sy zfs_commit_timeout_pct .
2218*15f0b8c3SMartin Matuska.
22193ff01b23SMartin Matuska.It Sy zil_nocacheflush Ns = Ns Sy 0 Ns | Ns 1 Pq int
22203ff01b23SMartin MatuskaDisable the cache flush commands that are normally sent to disk by
22213ff01b23SMartin Matuskathe ZIL after an LWB write has completed.
22223ff01b23SMartin MatuskaSetting this will cause ZIL corruption on power loss
22233ff01b23SMartin Matuskaif a volatile out-of-order write cache is enabled.
22243ff01b23SMartin Matuska.
22253ff01b23SMartin Matuska.It Sy zil_replay_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int
22263ff01b23SMartin MatuskaDisable intent logging replay.
22273ff01b23SMartin MatuskaCan be disabled for recovery from corrupted ZIL.
22283ff01b23SMartin Matuska.
2229dbd5678dSMartin Matuska.It Sy zil_slog_bulk Ns = Ns Sy 786432 Ns B Po 768 KiB Pc Pq u64
22303ff01b23SMartin MatuskaLimit SLOG write size per commit executed with synchronous priority.
22313ff01b23SMartin MatuskaAny writes above that will be executed with lower (asynchronous) priority
22323ff01b23SMartin Matuskato limit potential SLOG device abuse by single active ZIL writer.
22333ff01b23SMartin Matuska.
2234c03c5b1cSMartin Matuska.It Sy zfs_zil_saxattr Ns = Ns Sy 1 Ns | Ns 0 Pq int
2235c03c5b1cSMartin MatuskaSetting this tunable to zero disables ZIL logging of new
2236c03c5b1cSMartin Matuska.Sy xattr Ns = Ns Sy sa
2237c03c5b1cSMartin Matuskarecords if the
2238c03c5b1cSMartin Matuska.Sy org.openzfs:zilsaxattr
2239c03c5b1cSMartin Matuskafeature is enabled on the pool.
2240c03c5b1cSMartin MatuskaThis would only be necessary to work around bugs in the ZIL logging or replay
2241c03c5b1cSMartin Matuskacode for this record type.
2242c03c5b1cSMartin MatuskaThe tunable has no effect if the feature is disabled.
2243c03c5b1cSMartin Matuska.
2244be181ee2SMartin Matuska.It Sy zfs_embedded_slog_min_ms Ns = Ns Sy 64 Pq uint
22453ff01b23SMartin MatuskaUsually, one metaslab from each normal-class vdev is dedicated for use by
22463ff01b23SMartin Matuskathe ZIL to log synchronous writes.
22473ff01b23SMartin MatuskaHowever, if there are fewer than
22483ff01b23SMartin Matuska.Sy zfs_embedded_slog_min_ms
22493ff01b23SMartin Matuskametaslabs in the vdev, this functionality is disabled.
2250bb2d13b6SMartin MatuskaThis ensures that we don't set aside an unreasonable amount of space for the
2251bb2d13b6SMartin MatuskaZIL.
22523ff01b23SMartin Matuska.
2253be181ee2SMartin Matuska.It Sy zstd_earlyabort_pass Ns = Ns Sy 1 Pq uint
2254e3aa18adSMartin MatuskaWhether heuristic for detection of incompressible data with zstd levels >= 3
2255e3aa18adSMartin Matuskausing LZ4 and zstd-1 passes is enabled.
2256e3aa18adSMartin Matuska.
2257be181ee2SMartin Matuska.It Sy zstd_abort_size Ns = Ns Sy 131072 Pq uint
2258e3aa18adSMartin MatuskaMinimal uncompressed size (inclusive) of a record before the early abort
2259e3aa18adSMartin Matuskaheuristic will be attempted.
2260e3aa18adSMartin Matuska.
22613ff01b23SMartin Matuska.It Sy zio_deadman_log_all Ns = Ns Sy 0 Ns | Ns 1 Pq int
22623ff01b23SMartin MatuskaIf non-zero, the zio deadman will produce debugging messages
22633ff01b23SMartin Matuska.Pq see Sy zfs_dbgmsg_enable
22643ff01b23SMartin Matuskafor all zios, rather than only for leaf zios possessing a vdev.
22653ff01b23SMartin MatuskaThis is meant to be used by developers to gain
22663ff01b23SMartin Matuskadiagnostic information for hang conditions which don't involve a mutex
22673ff01b23SMartin Matuskaor other locking primitive: typically conditions in which a thread in
22683ff01b23SMartin Matuskathe zio pipeline is looping indefinitely.
22693ff01b23SMartin Matuska.
22703ff01b23SMartin Matuska.It Sy zio_slow_io_ms Ns = Ns Sy 30000 Ns ms Po 30 s Pc Pq int
22713ff01b23SMartin MatuskaWhen an I/O operation takes more than this much time to complete,
22723ff01b23SMartin Matuskait's marked as slow.
22733ff01b23SMartin MatuskaEach slow operation causes a delay zevent.
22743ff01b23SMartin MatuskaSlow I/O counters can be seen with
22753ff01b23SMartin Matuska.Nm zpool Cm status Fl s .
22763ff01b23SMartin Matuska.
22773ff01b23SMartin Matuska.It Sy zio_dva_throttle_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
22783ff01b23SMartin MatuskaThrottle block allocations in the I/O pipeline.
22793ff01b23SMartin MatuskaThis allows for dynamic allocation distribution when devices are imbalanced.
22803ff01b23SMartin MatuskaWhen enabled, the maximum number of pending allocations per top-level vdev
22813ff01b23SMartin Matuskais limited by
22823ff01b23SMartin Matuska.Sy zfs_vdev_queue_depth_pct .
22833ff01b23SMartin Matuska.
2284c03c5b1cSMartin Matuska.It Sy zfs_xattr_compat Ns = Ns 0 Ns | Ns 1 Pq int
2285c03c5b1cSMartin MatuskaControl the naming scheme used when setting new xattrs in the user namespace.
2286c03c5b1cSMartin MatuskaIf
2287c03c5b1cSMartin Matuska.Sy 0
2288c03c5b1cSMartin Matuska.Pq the default on Linux ,
2289c03c5b1cSMartin Matuskauser namespace xattr names are prefixed with the namespace, to be backwards
2290c03c5b1cSMartin Matuskacompatible with previous versions of ZFS on Linux.
2291c03c5b1cSMartin MatuskaIf
2292c03c5b1cSMartin Matuska.Sy 1
2293c03c5b1cSMartin Matuska.Pq the default on Fx ,
2294c03c5b1cSMartin Matuskauser namespace xattr names are not prefixed, to be backwards compatible with
2295c03c5b1cSMartin Matuskaprevious versions of ZFS on illumos and
2296c03c5b1cSMartin Matuska.Fx .
2297c03c5b1cSMartin Matuska.Pp
2298c03c5b1cSMartin MatuskaEither naming scheme can be read on this and future versions of ZFS, regardless
2299c03c5b1cSMartin Matuskaof this tunable, but legacy ZFS on illumos or
2300c03c5b1cSMartin Matuska.Fx
2301c03c5b1cSMartin Matuskaare unable to read user namespace xattrs written in the Linux format, and
2302c03c5b1cSMartin Matuskalegacy versions of ZFS on Linux are unable to read user namespace xattrs written
2303c03c5b1cSMartin Matuskain the legacy ZFS format.
2304c03c5b1cSMartin Matuska.Pp
2305c03c5b1cSMartin MatuskaAn existing xattr with the alternate naming scheme is removed when overwriting
2306c03c5b1cSMartin Matuskathe xattr so as to not accumulate duplicates.
2307c03c5b1cSMartin Matuska.
23083ff01b23SMartin Matuska.It Sy zio_requeue_io_start_cut_in_line Ns = Ns Sy 0 Ns | Ns 1 Pq int
23093ff01b23SMartin MatuskaPrioritize requeued I/O.
23103ff01b23SMartin Matuska.
23113ff01b23SMartin Matuska.It Sy zio_taskq_batch_pct Ns = Ns Sy 80 Ns % Pq uint
23123ff01b23SMartin MatuskaPercentage of online CPUs which will run a worker thread for I/O.
23133ff01b23SMartin MatuskaThese workers are responsible for I/O work such as compression and
23143ff01b23SMartin Matuskachecksum calculations.
23153ff01b23SMartin MatuskaFractional number of CPUs will be rounded down.
23163ff01b23SMartin Matuska.Pp
23173ff01b23SMartin MatuskaThe default value of
23183ff01b23SMartin Matuska.Sy 80%
23193ff01b23SMartin Matuskawas chosen to avoid using all CPUs which can result in
23203ff01b23SMartin Matuskalatency issues and inconsistent application performance,
23213ff01b23SMartin Matuskaespecially when slower compression and/or checksumming is enabled.
23223ff01b23SMartin Matuska.
23233ff01b23SMartin Matuska.It Sy zio_taskq_batch_tpq Ns = Ns Sy 0 Pq uint
23243ff01b23SMartin MatuskaNumber of worker threads per taskq.
23253ff01b23SMartin MatuskaLower values improve I/O ordering and CPU utilization,
23263ff01b23SMartin Matuskawhile higher reduces lock contention.
23273ff01b23SMartin Matuska.Pp
23283ff01b23SMartin MatuskaIf
23293ff01b23SMartin Matuska.Sy 0 ,
23303ff01b23SMartin Matuskagenerate a system-dependent value close to 6 threads per taskq.
23313ff01b23SMartin Matuska.
23323ff01b23SMartin Matuska.It Sy zvol_inhibit_dev Ns = Ns Sy 0 Ns | Ns 1 Pq uint
23333ff01b23SMartin MatuskaDo not create zvol device nodes.
23343ff01b23SMartin MatuskaThis may slightly improve startup time on
23353ff01b23SMartin Matuskasystems with a very large number of zvols.
23363ff01b23SMartin Matuska.
23373ff01b23SMartin Matuska.It Sy zvol_major Ns = Ns Sy 230 Pq uint
23383ff01b23SMartin MatuskaMajor number for zvol block devices.
23393ff01b23SMartin Matuska.
2340dbd5678dSMartin Matuska.It Sy zvol_max_discard_blocks Ns = Ns Sy 16384 Pq long
23413ff01b23SMartin MatuskaDiscard (TRIM) operations done on zvols will be done in batches of this
23423ff01b23SMartin Matuskamany blocks, where block size is determined by the
23433ff01b23SMartin Matuska.Sy volblocksize
23443ff01b23SMartin Matuskaproperty of a zvol.
23453ff01b23SMartin Matuska.
2346716fd348SMartin Matuska.It Sy zvol_prefetch_bytes Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint
23473ff01b23SMartin MatuskaWhen adding a zvol to the system, prefetch this many bytes
23483ff01b23SMartin Matuskafrom the start and end of the volume.
23493ff01b23SMartin MatuskaPrefetching these regions of the volume is desirable,
23503ff01b23SMartin Matuskabecause they are likely to be accessed immediately by
23513ff01b23SMartin Matuska.Xr blkid 8
23523ff01b23SMartin Matuskaor the kernel partitioner.
23533ff01b23SMartin Matuska.
23543ff01b23SMartin Matuska.It Sy zvol_request_sync Ns = Ns Sy 0 Ns | Ns 1 Pq uint
23553ff01b23SMartin MatuskaWhen processing I/O requests for a zvol, submit them synchronously.
23563ff01b23SMartin MatuskaThis effectively limits the queue depth to
23573ff01b23SMartin Matuska.Em 1
23583ff01b23SMartin Matuskafor each I/O submitter.
23593ff01b23SMartin MatuskaWhen unset, requests are handled asynchronously by a thread pool.
23603ff01b23SMartin MatuskaThe number of requests which can be handled concurrently is controlled by
23613ff01b23SMartin Matuska.Sy zvol_threads .
23621f1e2261SMartin Matuska.Sy zvol_request_sync
23631f1e2261SMartin Matuskais ignored when running on a kernel that supports block multiqueue
23641f1e2261SMartin Matuska.Pq Li blk-mq .
23653ff01b23SMartin Matuska.
23661f1e2261SMartin Matuska.It Sy zvol_threads Ns = Ns Sy 0 Pq uint
23671f1e2261SMartin MatuskaThe number of system wide threads to use for processing zvol block IOs.
23681f1e2261SMartin MatuskaIf
23691f1e2261SMartin Matuska.Sy 0
23701f1e2261SMartin Matuska(the default) then internally set
23711f1e2261SMartin Matuska.Sy zvol_threads
23721f1e2261SMartin Matuskato the number of CPUs present or 32 (whichever is greater).
23731f1e2261SMartin Matuska.
23741f1e2261SMartin Matuska.It Sy zvol_blk_mq_threads Ns = Ns Sy 0 Pq uint
23751f1e2261SMartin MatuskaThe number of threads per zvol to use for queuing IO requests.
23761f1e2261SMartin MatuskaThis parameter will only appear if your kernel supports
23771f1e2261SMartin Matuska.Li blk-mq
23781f1e2261SMartin Matuskaand is only read and assigned to a zvol at zvol load time.
23791f1e2261SMartin MatuskaIf
23801f1e2261SMartin Matuska.Sy 0
23811f1e2261SMartin Matuska(the default) then internally set
23821f1e2261SMartin Matuska.Sy zvol_blk_mq_threads
23831f1e2261SMartin Matuskato the number of CPUs present.
23841f1e2261SMartin Matuska.
23851f1e2261SMartin Matuska.It Sy zvol_use_blk_mq Ns = Ns Sy 0 Ns | Ns 1 Pq uint
23861f1e2261SMartin MatuskaSet to
23871f1e2261SMartin Matuska.Sy 1
23881f1e2261SMartin Matuskato use the
23891f1e2261SMartin Matuska.Li blk-mq
23901f1e2261SMartin MatuskaAPI for zvols.
23911f1e2261SMartin MatuskaSet to
23921f1e2261SMartin Matuska.Sy 0
23931f1e2261SMartin Matuska(the default) to use the legacy zvol APIs.
23941f1e2261SMartin MatuskaThis setting can give better or worse zvol performance depending on
23951f1e2261SMartin Matuskathe workload.
23961f1e2261SMartin MatuskaThis parameter will only appear if your kernel supports
23971f1e2261SMartin Matuska.Li blk-mq
23981f1e2261SMartin Matuskaand is only read and assigned to a zvol at zvol load time.
23991f1e2261SMartin Matuska.
24001f1e2261SMartin Matuska.It Sy zvol_blk_mq_blocks_per_thread Ns = Ns Sy 8 Pq uint
24011f1e2261SMartin MatuskaIf
24021f1e2261SMartin Matuska.Sy zvol_use_blk_mq
24031f1e2261SMartin Matuskais enabled, then process this number of
24041f1e2261SMartin Matuska.Sy volblocksize Ns -sized blocks per zvol thread.
24051f1e2261SMartin MatuskaThis tunable can be use to favor better performance for zvol reads (lower
24061f1e2261SMartin Matuskavalues) or writes (higher values).
24071f1e2261SMartin MatuskaIf set to
24081f1e2261SMartin Matuska.Sy 0 ,
24091f1e2261SMartin Matuskathen the zvol layer will process the maximum number of blocks
24101f1e2261SMartin Matuskaper thread that it can.
24111f1e2261SMartin MatuskaThis parameter will only appear if your kernel supports
24121f1e2261SMartin Matuska.Li blk-mq
24131f1e2261SMartin Matuskaand is only applied at each zvol's load time.
24141f1e2261SMartin Matuska.
24151f1e2261SMartin Matuska.It Sy zvol_blk_mq_queue_depth Ns = Ns Sy 0 Pq uint
24161f1e2261SMartin MatuskaThe queue_depth value for the zvol
24171f1e2261SMartin Matuska.Li blk-mq
24181f1e2261SMartin Matuskainterface.
24191f1e2261SMartin MatuskaThis parameter will only appear if your kernel supports
24201f1e2261SMartin Matuska.Li blk-mq
24211f1e2261SMartin Matuskaand is only applied at each zvol's load time.
24221f1e2261SMartin MatuskaIf
24231f1e2261SMartin Matuska.Sy 0
24241f1e2261SMartin Matuska(the default) then use the kernel's default queue depth.
24251f1e2261SMartin MatuskaValues are clamped to the kernel's
24261f1e2261SMartin Matuska.Dv BLKDEV_MIN_RQ
24271f1e2261SMartin Matuskaand
24281f1e2261SMartin Matuska.Dv BLKDEV_MAX_RQ Ns / Ns Dv BLKDEV_DEFAULT_RQ
24291f1e2261SMartin Matuskalimits.
24303ff01b23SMartin Matuska.
24313ff01b23SMartin Matuska.It Sy zvol_volmode Ns = Ns Sy 1 Pq uint
24323ff01b23SMartin MatuskaDefines zvol block devices behaviour when
24333ff01b23SMartin Matuska.Sy volmode Ns = Ns Sy default :
24343ff01b23SMartin Matuska.Bl -tag -compact -offset 4n -width "a"
24353ff01b23SMartin Matuska.It Sy 1
24363ff01b23SMartin Matuska.No equivalent to Sy full
24373ff01b23SMartin Matuska.It Sy 2
24383ff01b23SMartin Matuska.No equivalent to Sy dev
24393ff01b23SMartin Matuska.It Sy 3
24403ff01b23SMartin Matuska.No equivalent to Sy none
24413ff01b23SMartin Matuska.El
2442dbd5678dSMartin Matuska.
2443dbd5678dSMartin Matuska.It Sy zvol_enforce_quotas Ns = Ns Sy 0 Ns | Ns 1 Pq uint
2444dbd5678dSMartin MatuskaEnable strict ZVOL quota enforcement.
2445dbd5678dSMartin MatuskaThe strict quota enforcement may have a performance impact.
24463ff01b23SMartin Matuska.El
24473ff01b23SMartin Matuska.
24483ff01b23SMartin Matuska.Sh ZFS I/O SCHEDULER
24493ff01b23SMartin MatuskaZFS issues I/O operations to leaf vdevs to satisfy and complete I/O operations.
24503ff01b23SMartin MatuskaThe scheduler determines when and in what order those operations are issued.
24513ff01b23SMartin MatuskaThe scheduler divides operations into five I/O classes,
24523ff01b23SMartin Matuskaprioritized in the following order: sync read, sync write, async read,
24533ff01b23SMartin Matuskaasync write, and scrub/resilver.
24543ff01b23SMartin MatuskaEach queue defines the minimum and maximum number of concurrent operations
24553ff01b23SMartin Matuskathat may be issued to the device.
24563ff01b23SMartin MatuskaIn addition, the device has an aggregate maximum,
24573ff01b23SMartin Matuska.Sy zfs_vdev_max_active .
24583ff01b23SMartin MatuskaNote that the sum of the per-queue minima must not exceed the aggregate maximum.
24593ff01b23SMartin MatuskaIf the sum of the per-queue maxima exceeds the aggregate maximum,
24603ff01b23SMartin Matuskathen the number of active operations may reach
24613ff01b23SMartin Matuska.Sy zfs_vdev_max_active ,
24623ff01b23SMartin Matuskain which case no further operations will be issued,
24633ff01b23SMartin Matuskaregardless of whether all per-queue minima have been met.
24643ff01b23SMartin Matuska.Pp
24653ff01b23SMartin MatuskaFor many physical devices, throughput increases with the number of
24663ff01b23SMartin Matuskaconcurrent operations, but latency typically suffers.
24673ff01b23SMartin MatuskaFurthermore, physical devices typically have a limit
24683ff01b23SMartin Matuskaat which more concurrent operations have no
24693ff01b23SMartin Matuskaeffect on throughput or can actually cause it to decrease.
24703ff01b23SMartin Matuska.Pp
24713ff01b23SMartin MatuskaThe scheduler selects the next operation to issue by first looking for an
24723ff01b23SMartin MatuskaI/O class whose minimum has not been satisfied.
24733ff01b23SMartin MatuskaOnce all are satisfied and the aggregate maximum has not been hit,
24743ff01b23SMartin Matuskathe scheduler looks for classes whose maximum has not been satisfied.
24753ff01b23SMartin MatuskaIteration through the I/O classes is done in the order specified above.
24763ff01b23SMartin MatuskaNo further operations are issued
24773ff01b23SMartin Matuskaif the aggregate maximum number of concurrent operations has been hit,
2478bb2d13b6SMartin Matuskaor if there are no operations queued for an I/O class that has not hit its
2479bb2d13b6SMartin Matuskamaximum.
24803ff01b23SMartin MatuskaEvery time an I/O operation is queued or an operation completes,
24813ff01b23SMartin Matuskathe scheduler looks for new operations to issue.
24823ff01b23SMartin Matuska.Pp
24833ff01b23SMartin MatuskaIn general, smaller
24843ff01b23SMartin Matuska.Sy max_active Ns s
24853ff01b23SMartin Matuskawill lead to lower latency of synchronous operations.
24863ff01b23SMartin MatuskaLarger
24873ff01b23SMartin Matuska.Sy max_active Ns s
24883ff01b23SMartin Matuskamay lead to higher overall throughput, depending on underlying storage.
24893ff01b23SMartin Matuska.Pp
24903ff01b23SMartin MatuskaThe ratio of the queues'
24913ff01b23SMartin Matuska.Sy max_active Ns s
24923ff01b23SMartin Matuskadetermines the balance of performance between reads, writes, and scrubs.
24933ff01b23SMartin MatuskaFor example, increasing
24943ff01b23SMartin Matuska.Sy zfs_vdev_scrub_max_active
24953ff01b23SMartin Matuskawill cause the scrub or resilver to complete more quickly,
24963ff01b23SMartin Matuskabut reads and writes to have higher latency and lower throughput.
24973ff01b23SMartin Matuska.Pp
24983ff01b23SMartin MatuskaAll I/O classes have a fixed maximum number of outstanding operations,
24993ff01b23SMartin Matuskaexcept for the async write class.
25003ff01b23SMartin MatuskaAsynchronous writes represent the data that is committed to stable storage
25013ff01b23SMartin Matuskaduring the syncing stage for transaction groups.
25023ff01b23SMartin MatuskaTransaction groups enter the syncing state periodically,
25033ff01b23SMartin Matuskaso the number of queued async writes will quickly burst up
25043ff01b23SMartin Matuskaand then bleed down to zero.
25053ff01b23SMartin MatuskaRather than servicing them as quickly as possible,
25063ff01b23SMartin Matuskathe I/O scheduler changes the maximum number of active async write operations
25073ff01b23SMartin Matuskaaccording to the amount of dirty data in the pool.
25083ff01b23SMartin MatuskaSince both throughput and latency typically increase with the number of
25093ff01b23SMartin Matuskaconcurrent operations issued to physical devices, reducing the
2510bb2d13b6SMartin Matuskaburstiness in the number of simultaneous operations also stabilizes the
2511bb2d13b6SMartin Matuskaresponse time of operations from other queues, in particular synchronous ones.
25123ff01b23SMartin MatuskaIn broad strokes, the I/O scheduler will issue more concurrent operations
2513bb2d13b6SMartin Matuskafrom the async write queue as there is more dirty data in the pool.
25143ff01b23SMartin Matuska.
25153ff01b23SMartin Matuska.Ss Async Writes
25163ff01b23SMartin MatuskaThe number of concurrent operations issued for the async write I/O class
25173ff01b23SMartin Matuskafollows a piece-wise linear function defined by a few adjustable points:
25183ff01b23SMartin Matuska.Bd -literal
25193ff01b23SMartin Matuska       |              o---------| <-- \fBzfs_vdev_async_write_max_active\fP
25203ff01b23SMartin Matuska  ^    |             /^         |
25213ff01b23SMartin Matuska  |    |            / |         |
25223ff01b23SMartin Matuskaactive |           /  |         |
25233ff01b23SMartin Matuska I/O   |          /   |         |
25243ff01b23SMartin Matuskacount  |         /    |         |
25253ff01b23SMartin Matuska       |        /     |         |
25263ff01b23SMartin Matuska       |-------o      |         | <-- \fBzfs_vdev_async_write_min_active\fP
25273ff01b23SMartin Matuska      0|_______^______|_________|
25283ff01b23SMartin Matuska       0%      |      |       100% of \fBzfs_dirty_data_max\fP
25293ff01b23SMartin Matuska               |      |
25303ff01b23SMartin Matuska               |      `-- \fBzfs_vdev_async_write_active_max_dirty_percent\fP
25313ff01b23SMartin Matuska               `--------- \fBzfs_vdev_async_write_active_min_dirty_percent\fP
25323ff01b23SMartin Matuska.Ed
25333ff01b23SMartin Matuska.Pp
25343ff01b23SMartin MatuskaUntil the amount of dirty data exceeds a minimum percentage of the dirty
25353ff01b23SMartin Matuskadata allowed in the pool, the I/O scheduler will limit the number of
25363ff01b23SMartin Matuskaconcurrent operations to the minimum.
25373ff01b23SMartin MatuskaAs that threshold is crossed, the number of concurrent operations issued
25383ff01b23SMartin Matuskaincreases linearly to the maximum at the specified maximum percentage
25393ff01b23SMartin Matuskaof the dirty data allowed in the pool.
25403ff01b23SMartin Matuska.Pp
25413ff01b23SMartin MatuskaIdeally, the amount of dirty data on a busy pool will stay in the sloped
25423ff01b23SMartin Matuskapart of the function between
25433ff01b23SMartin Matuska.Sy zfs_vdev_async_write_active_min_dirty_percent
25443ff01b23SMartin Matuskaand
25453ff01b23SMartin Matuska.Sy zfs_vdev_async_write_active_max_dirty_percent .
25463ff01b23SMartin MatuskaIf it exceeds the maximum percentage,
25473ff01b23SMartin Matuskathis indicates that the rate of incoming data is
25483ff01b23SMartin Matuskagreater than the rate that the backend storage can handle.
25493ff01b23SMartin MatuskaIn this case, we must further throttle incoming writes,
25503ff01b23SMartin Matuskaas described in the next section.
25513ff01b23SMartin Matuska.
25523ff01b23SMartin Matuska.Sh ZFS TRANSACTION DELAY
25533ff01b23SMartin MatuskaWe delay transactions when we've determined that the backend storage
25543ff01b23SMartin Matuskaisn't able to accommodate the rate of incoming writes.
25553ff01b23SMartin Matuska.Pp
25563ff01b23SMartin MatuskaIf there is already a transaction waiting, we delay relative to when
25573ff01b23SMartin Matuskathat transaction will finish waiting.
25583ff01b23SMartin MatuskaThis way the calculated delay time
25593ff01b23SMartin Matuskais independent of the number of threads concurrently executing transactions.
25603ff01b23SMartin Matuska.Pp
25613ff01b23SMartin MatuskaIf we are the only waiter, wait relative to when the transaction started,
25623ff01b23SMartin Matuskarather than the current time.
25633ff01b23SMartin MatuskaThis credits the transaction for "time already served",
25643ff01b23SMartin Matuskae.g. reading indirect blocks.
25653ff01b23SMartin Matuska.Pp
25663ff01b23SMartin MatuskaThe minimum time for a transaction to take is calculated as
2567e92ffd9bSMartin Matuska.D1 min_time = min( Ns Sy zfs_delay_scale No \(mu Po Sy dirty No \- Sy min Pc / Po Sy max No \- Sy dirty Pc , 100ms)
25683ff01b23SMartin Matuska.Pp
25693ff01b23SMartin MatuskaThe delay has two degrees of freedom that can be adjusted via tunables.
25703ff01b23SMartin MatuskaThe percentage of dirty data at which we start to delay is defined by
25713ff01b23SMartin Matuska.Sy zfs_delay_min_dirty_percent .
25723ff01b23SMartin MatuskaThis should typically be at or above
25733ff01b23SMartin Matuska.Sy zfs_vdev_async_write_active_max_dirty_percent ,
25743ff01b23SMartin Matuskaso that we only start to delay after writing at full speed
25753ff01b23SMartin Matuskahas failed to keep up with the incoming write rate.
25763ff01b23SMartin MatuskaThe scale of the curve is defined by
25773ff01b23SMartin Matuska.Sy zfs_delay_scale .
2578bb2d13b6SMartin MatuskaRoughly speaking, this variable determines the amount of delay at the midpoint
2579bb2d13b6SMartin Matuskaof the curve.
25803ff01b23SMartin Matuska.Bd -literal
25813ff01b23SMartin Matuskadelay
25823ff01b23SMartin Matuska 10ms +-------------------------------------------------------------*+
25833ff01b23SMartin Matuska      |                                                             *|
25843ff01b23SMartin Matuska  9ms +                                                             *+
25853ff01b23SMartin Matuska      |                                                             *|
25863ff01b23SMartin Matuska  8ms +                                                             *+
25873ff01b23SMartin Matuska      |                                                            * |
25883ff01b23SMartin Matuska  7ms +                                                            * +
25893ff01b23SMartin Matuska      |                                                            * |
25903ff01b23SMartin Matuska  6ms +                                                            * +
25913ff01b23SMartin Matuska      |                                                            * |
25923ff01b23SMartin Matuska  5ms +                                                           *  +
25933ff01b23SMartin Matuska      |                                                           *  |
25943ff01b23SMartin Matuska  4ms +                                                           *  +
25953ff01b23SMartin Matuska      |                                                           *  |
25963ff01b23SMartin Matuska  3ms +                                                          *   +
25973ff01b23SMartin Matuska      |                                                          *   |
25983ff01b23SMartin Matuska  2ms +                                              (midpoint) *    +
25993ff01b23SMartin Matuska      |                                                  |    **     |
26003ff01b23SMartin Matuska  1ms +                                                  v ***       +
26013ff01b23SMartin Matuska      |             \fBzfs_delay_scale\fP ---------->     ********         |
26023ff01b23SMartin Matuska    0 +-------------------------------------*********----------------+
26033ff01b23SMartin Matuska      0%                    <- \fBzfs_dirty_data_max\fP ->               100%
26043ff01b23SMartin Matuska.Ed
26053ff01b23SMartin Matuska.Pp
26063ff01b23SMartin MatuskaNote, that since the delay is added to the outstanding time remaining on the
26073ff01b23SMartin Matuskamost recent transaction it's effectively the inverse of IOPS.
26083ff01b23SMartin MatuskaHere, the midpoint of
26093ff01b23SMartin Matuska.Em 500 us
26103ff01b23SMartin Matuskatranslates to
26113ff01b23SMartin Matuska.Em 2000 IOPS .
26123ff01b23SMartin MatuskaThe shape of the curve
26133ff01b23SMartin Matuskawas chosen such that small changes in the amount of accumulated dirty data
26143ff01b23SMartin Matuskain the first three quarters of the curve yield relatively small differences
26153ff01b23SMartin Matuskain the amount of delay.
26163ff01b23SMartin Matuska.Pp
26173ff01b23SMartin MatuskaThe effects can be easier to understand when the amount of delay is
26183ff01b23SMartin Matuskarepresented on a logarithmic scale:
26193ff01b23SMartin Matuska.Bd -literal
26203ff01b23SMartin Matuskadelay
26213ff01b23SMartin Matuska100ms +-------------------------------------------------------------++
26223ff01b23SMartin Matuska      +                                                              +
26233ff01b23SMartin Matuska      |                                                              |
26243ff01b23SMartin Matuska      +                                                             *+
26253ff01b23SMartin Matuska 10ms +                                                             *+
26263ff01b23SMartin Matuska      +                                                           ** +
26273ff01b23SMartin Matuska      |                                              (midpoint)  **  |
26283ff01b23SMartin Matuska      +                                                  |     **    +
26293ff01b23SMartin Matuska  1ms +                                                  v ****      +
26303ff01b23SMartin Matuska      +             \fBzfs_delay_scale\fP ---------->        *****         +
26313ff01b23SMartin Matuska      |                                             ****             |
26323ff01b23SMartin Matuska      +                                          ****                +
26333ff01b23SMartin Matuska100us +                                        **                    +
26343ff01b23SMartin Matuska      +                                       *                      +
26353ff01b23SMartin Matuska      |                                      *                       |
26363ff01b23SMartin Matuska      +                                     *                        +
26373ff01b23SMartin Matuska 10us +                                     *                        +
26383ff01b23SMartin Matuska      +                                                              +
26393ff01b23SMartin Matuska      |                                                              |
26403ff01b23SMartin Matuska      +                                                              +
26413ff01b23SMartin Matuska      +--------------------------------------------------------------+
26423ff01b23SMartin Matuska      0%                    <- \fBzfs_dirty_data_max\fP ->               100%
26433ff01b23SMartin Matuska.Ed
26443ff01b23SMartin Matuska.Pp
26453ff01b23SMartin MatuskaNote here that only as the amount of dirty data approaches its limit does
26463ff01b23SMartin Matuskathe delay start to increase rapidly.
26473ff01b23SMartin MatuskaThe goal of a properly tuned system should be to keep the amount of dirty data
26483ff01b23SMartin Matuskaout of that range by first ensuring that the appropriate limits are set
26493ff01b23SMartin Matuskafor the I/O scheduler to reach optimal throughput on the back-end storage,
26503ff01b23SMartin Matuskaand then by changing the value of
26513ff01b23SMartin Matuska.Sy zfs_delay_scale
26523ff01b23SMartin Matuskato increase the steepness of the curve.
2653