xref: /freebsd/sys/contrib/openzfs/man/man4/zfs.4 (revision c7046f76c2c027b00c0e6ba57cfd28f1a78f5e23)
1.\"
2.\" Copyright (c) 2013 by Turbo Fredriksson <turbo@bayour.com>. All rights reserved.
3.\" Copyright (c) 2019, 2021 by Delphix. All rights reserved.
4.\" Copyright (c) 2019 Datto Inc.
5.\" The contents of this file are subject to the terms of the Common Development
6.\" and Distribution License (the "License").  You may not use this file except
7.\" in compliance with the License. You can obtain a copy of the license at
8.\" usr/src/OPENSOLARIS.LICENSE or https://opensource.org/licenses/CDDL-1.0.
9.\"
10.\" See the License for the specific language governing permissions and
11.\" limitations under the License. When distributing Covered Code, include this
12.\" CDDL HEADER in each file and include the License file at
13.\" usr/src/OPENSOLARIS.LICENSE.  If applicable, add the following below this
14.\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your
15.\" own identifying information:
16.\" Portions Copyright [yyyy] [name of copyright owner]
17.\"
18.Dd June 1, 2021
19.Dt ZFS 4
20.Os
21.
22.Sh NAME
23.Nm zfs
24.Nd tuning of the ZFS kernel module
25.
26.Sh DESCRIPTION
27The ZFS module supports these parameters:
28.Bl -tag -width Ds
29.It Sy dbuf_cache_max_bytes Ns = Ns Sy ULONG_MAX Ns B Pq ulong
30Maximum size in bytes of the dbuf cache.
31The target size is determined by the MIN versus
32.No 1/2^ Ns Sy dbuf_cache_shift Pq 1/32nd
33of the target ARC size.
34The behavior of the dbuf cache and its associated settings
35can be observed via the
36.Pa /proc/spl/kstat/zfs/dbufstats
37kstat.
38.
39.It Sy dbuf_metadata_cache_max_bytes Ns = Ns Sy ULONG_MAX Ns B Pq ulong
40Maximum size in bytes of the metadata dbuf cache.
41The target size is determined by the MIN versus
42.No 1/2^ Ns Sy dbuf_metadata_cache_shift Pq 1/64th
43of the target ARC size.
44The behavior of the metadata dbuf cache and its associated settings
45can be observed via the
46.Pa /proc/spl/kstat/zfs/dbufstats
47kstat.
48.
49.It Sy dbuf_cache_hiwater_pct Ns = Ns Sy 10 Ns % Pq uint
50The percentage over
51.Sy dbuf_cache_max_bytes
52when dbufs must be evicted directly.
53.
54.It Sy dbuf_cache_lowater_pct Ns = Ns Sy 10 Ns % Pq uint
55The percentage below
56.Sy dbuf_cache_max_bytes
57when the evict thread stops evicting dbufs.
58.
59.It Sy dbuf_cache_shift Ns = Ns Sy 5 Pq int
60Set the size of the dbuf cache
61.Pq Sy dbuf_cache_max_bytes
62to a log2 fraction of the target ARC size.
63.
64.It Sy dbuf_metadata_cache_shift Ns = Ns Sy 6 Pq int
65Set the size of the dbuf metadata cache
66.Pq Sy dbuf_metadata_cache_max_bytes
67to a log2 fraction of the target ARC size.
68.
69.It Sy dmu_object_alloc_chunk_shift Ns = Ns Sy 7 Po 128 Pc Pq int
70dnode slots allocated in a single operation as a power of 2.
71The default value minimizes lock contention for the bulk operation performed.
72.
73.It Sy dmu_prefetch_max Ns = Ns Sy 134217728 Ns B Po 128 MiB Pc Pq int
74Limit the amount we can prefetch with one call to this amount in bytes.
75This helps to limit the amount of memory that can be used by prefetching.
76.
77.It Sy ignore_hole_birth Pq int
78Alias for
79.Sy send_holes_without_birth_time .
80.
81.It Sy l2arc_feed_again Ns = Ns Sy 1 Ns | Ns 0 Pq int
82Turbo L2ARC warm-up.
83When the L2ARC is cold the fill interval will be set as fast as possible.
84.
85.It Sy l2arc_feed_min_ms Ns = Ns Sy 200 Pq ulong
86Min feed interval in milliseconds.
87Requires
88.Sy l2arc_feed_again Ns = Ns Ar 1
89and only applicable in related situations.
90.
91.It Sy l2arc_feed_secs Ns = Ns Sy 1 Pq ulong
92Seconds between L2ARC writing.
93.
94.It Sy l2arc_headroom Ns = Ns Sy 2 Pq ulong
95How far through the ARC lists to search for L2ARC cacheable content,
96expressed as a multiplier of
97.Sy l2arc_write_max .
98ARC persistence across reboots can be achieved with persistent L2ARC
99by setting this parameter to
100.Sy 0 ,
101allowing the full length of ARC lists to be searched for cacheable content.
102.
103.It Sy l2arc_headroom_boost Ns = Ns Sy 200 Ns % Pq ulong
104Scales
105.Sy l2arc_headroom
106by this percentage when L2ARC contents are being successfully compressed
107before writing.
108A value of
109.Sy 100
110disables this feature.
111.
112.It Sy l2arc_exclude_special Ns = Ns Sy 0 Ns | Ns 1 Pq int
113Controls whether buffers present on special vdevs are eligible for caching
114into L2ARC.
115If set to 1, exclude dbufs on special vdevs from being cached to L2ARC.
116.
117.It Sy l2arc_mfuonly Ns = Ns Sy 0 Ns | Ns 1 Pq  int
118Controls whether only MFU metadata and data are cached from ARC into L2ARC.
119This may be desired to avoid wasting space on L2ARC when reading/writing large
120amounts of data that are not expected to be accessed more than once.
121.Pp
122The default is off,
123meaning both MRU and MFU data and metadata are cached.
124When turning off this feature, some MRU buffers will still be present
125in ARC and eventually cached on L2ARC.
126.No If Sy l2arc_noprefetch Ns = Ns Sy 0 ,
127some prefetched buffers will be cached to L2ARC, and those might later
128transition to MRU, in which case the
129.Sy l2arc_mru_asize No arcstat will not be Sy 0 .
130.Pp
131Regardless of
132.Sy l2arc_noprefetch ,
133some MFU buffers might be evicted from ARC,
134accessed later on as prefetches and transition to MRU as prefetches.
135If accessed again they are counted as MRU and the
136.Sy l2arc_mru_asize No arcstat will not be Sy 0 .
137.Pp
138The ARC status of L2ARC buffers when they were first cached in
139L2ARC can be seen in the
140.Sy l2arc_mru_asize , Sy l2arc_mfu_asize , No and Sy l2arc_prefetch_asize
141arcstats when importing the pool or onlining a cache
142device if persistent L2ARC is enabled.
143.Pp
144The
145.Sy evict_l2_eligible_mru
146arcstat does not take into account if this option is enabled as the information
147provided by the
148.Sy evict_l2_eligible_m[rf]u
149arcstats can be used to decide if toggling this option is appropriate
150for the current workload.
151.
152.It Sy l2arc_meta_percent Ns = Ns Sy 33 Ns % Pq int
153Percent of ARC size allowed for L2ARC-only headers.
154Since L2ARC buffers are not evicted on memory pressure,
155too many headers on a system with an irrationally large L2ARC
156can render it slow or unusable.
157This parameter limits L2ARC writes and rebuilds to achieve the target.
158.
159.It Sy l2arc_trim_ahead Ns = Ns Sy 0 Ns % Pq ulong
160Trims ahead of the current write size
161.Pq Sy l2arc_write_max
162on L2ARC devices by this percentage of write size if we have filled the device.
163If set to
164.Sy 100
165we TRIM twice the space required to accommodate upcoming writes.
166A minimum of
167.Sy 64 MiB
168will be trimmed.
169It also enables TRIM of the whole L2ARC device upon creation
170or addition to an existing pool or if the header of the device is
171invalid upon importing a pool or onlining a cache device.
172A value of
173.Sy 0
174disables TRIM on L2ARC altogether and is the default as it can put significant
175stress on the underlying storage devices.
176This will vary depending of how well the specific device handles these commands.
177.
178.It Sy l2arc_noprefetch Ns = Ns Sy 1 Ns | Ns 0 Pq int
179Do not write buffers to L2ARC if they were prefetched but not used by
180applications.
181In case there are prefetched buffers in L2ARC and this option
182is later set, we do not read the prefetched buffers from L2ARC.
183Unsetting this option is useful for caching sequential reads from the
184disks to L2ARC and serve those reads from L2ARC later on.
185This may be beneficial in case the L2ARC device is significantly faster
186in sequential reads than the disks of the pool.
187.Pp
188Use
189.Sy 1
190to disable and
191.Sy 0
192to enable caching/reading prefetches to/from L2ARC.
193.
194.It Sy l2arc_norw Ns = Ns Sy 0 Ns | Ns 1 Pq int
195No reads during writes.
196.
197.It Sy l2arc_write_boost Ns = Ns Sy 8388608 Ns B Po 8 MiB Pc Pq ulong
198Cold L2ARC devices will have
199.Sy l2arc_write_max
200increased by this amount while they remain cold.
201.
202.It Sy l2arc_write_max Ns = Ns Sy 8388608 Ns B Po 8 MiB Pc Pq ulong
203Max write bytes per interval.
204.
205.It Sy l2arc_rebuild_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
206Rebuild the L2ARC when importing a pool (persistent L2ARC).
207This can be disabled if there are problems importing a pool
208or attaching an L2ARC device (e.g. the L2ARC device is slow
209in reading stored log metadata, or the metadata
210has become somehow fragmented/unusable).
211.
212.It Sy l2arc_rebuild_blocks_min_l2size Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq ulong
213Mininum size of an L2ARC device required in order to write log blocks in it.
214The log blocks are used upon importing the pool to rebuild the persistent L2ARC.
215.Pp
216For L2ARC devices less than 1 GiB, the amount of data
217.Fn l2arc_evict
218evicts is significant compared to the amount of restored L2ARC data.
219In this case, do not write log blocks in L2ARC in order not to waste space.
220.
221.It Sy metaslab_aliquot Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq ulong
222Metaslab granularity, in bytes.
223This is roughly similar to what would be referred to as the "stripe size"
224in traditional RAID arrays.
225In normal operation, ZFS will try to write this amount of data to each disk
226before moving on to the next top-level vdev.
227.
228.It Sy metaslab_bias_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
229Enable metaslab group biasing based on their vdevs' over- or under-utilization
230relative to the pool.
231.
232.It Sy metaslab_force_ganging Ns = Ns Sy 16777217 Ns B Po 16 MiB + 1 B Pc Pq ulong
233Make some blocks above a certain size be gang blocks.
234This option is used by the test suite to facilitate testing.
235.
236.It Sy zfs_history_output_max Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq int
237When attempting to log an output nvlist of an ioctl in the on-disk history,
238the output will not be stored if it is larger than this size (in bytes).
239This must be less than
240.Sy DMU_MAX_ACCESS Pq 64 MiB .
241This applies primarily to
242.Fn zfs_ioc_channel_program Pq cf. Xr zfs-program 8 .
243.
244.It Sy zfs_keep_log_spacemaps_at_export Ns = Ns Sy 0 Ns | Ns 1 Pq int
245Prevent log spacemaps from being destroyed during pool exports and destroys.
246.
247.It Sy zfs_metaslab_segment_weight_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
248Enable/disable segment-based metaslab selection.
249.
250.It Sy zfs_metaslab_switch_threshold Ns = Ns Sy 2 Pq int
251When using segment-based metaslab selection, continue allocating
252from the active metaslab until this option's
253worth of buckets have been exhausted.
254.
255.It Sy metaslab_debug_load Ns = Ns Sy 0 Ns | Ns 1 Pq int
256Load all metaslabs during pool import.
257.
258.It Sy metaslab_debug_unload Ns = Ns Sy 0 Ns | Ns 1 Pq int
259Prevent metaslabs from being unloaded.
260.
261.It Sy metaslab_fragmentation_factor_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
262Enable use of the fragmentation metric in computing metaslab weights.
263.
264.It Sy metaslab_df_max_search Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq int
265Maximum distance to search forward from the last offset.
266Without this limit, fragmented pools can see
267.Em >100`000
268iterations and
269.Fn metaslab_block_picker
270becomes the performance limiting factor on high-performance storage.
271.Pp
272With the default setting of
273.Sy 16 MiB ,
274we typically see less than
275.Em 500
276iterations, even with very fragmented
277.Sy ashift Ns = Ns Sy 9
278pools.
279The maximum number of iterations possible is
280.Sy metaslab_df_max_search / 2^(ashift+1) .
281With the default setting of
282.Sy 16 MiB
283this is
284.Em 16*1024 Pq with Sy ashift Ns = Ns Sy 9
285or
286.Em 2*1024 Pq with Sy ashift Ns = Ns Sy 12 .
287.
288.It Sy metaslab_df_use_largest_segment Ns = Ns Sy 0 Ns | Ns 1 Pq int
289If not searching forward (due to
290.Sy metaslab_df_max_search , metaslab_df_free_pct ,
291.No or Sy metaslab_df_alloc_threshold ) ,
292this tunable controls which segment is used.
293If set, we will use the largest free segment.
294If unset, we will use a segment of at least the requested size.
295.
296.It Sy zfs_metaslab_max_size_cache_sec Ns = Ns Sy 3600 Ns s Po 1 hour Pc Pq ulong
297When we unload a metaslab, we cache the size of the largest free chunk.
298We use that cached size to determine whether or not to load a metaslab
299for a given allocation.
300As more frees accumulate in that metaslab while it's unloaded,
301the cached max size becomes less and less accurate.
302After a number of seconds controlled by this tunable,
303we stop considering the cached max size and start
304considering only the histogram instead.
305.
306.It Sy zfs_metaslab_mem_limit Ns = Ns Sy 25 Ns % Pq int
307When we are loading a new metaslab, we check the amount of memory being used
308to store metaslab range trees.
309If it is over a threshold, we attempt to unload the least recently used metaslab
310to prevent the system from clogging all of its memory with range trees.
311This tunable sets the percentage of total system memory that is the threshold.
312.
313.It Sy zfs_metaslab_try_hard_before_gang Ns = Ns Sy 0 Ns | Ns 1 Pq int
314.Bl -item -compact
315.It
316If unset, we will first try normal allocation.
317.It
318If that fails then we will do a gang allocation.
319.It
320If that fails then we will do a "try hard" gang allocation.
321.It
322If that fails then we will have a multi-layer gang block.
323.El
324.Pp
325.Bl -item -compact
326.It
327If set, we will first try normal allocation.
328.It
329If that fails then we will do a "try hard" allocation.
330.It
331If that fails we will do a gang allocation.
332.It
333If that fails we will do a "try hard" gang allocation.
334.It
335If that fails then we will have a multi-layer gang block.
336.El
337.
338.It Sy zfs_metaslab_find_max_tries Ns = Ns Sy 100 Pq int
339When not trying hard, we only consider this number of the best metaslabs.
340This improves performance, especially when there are many metaslabs per vdev
341and the allocation can't actually be satisfied
342(so we would otherwise iterate all metaslabs).
343.
344.It Sy zfs_vdev_default_ms_count Ns = Ns Sy 200 Pq int
345When a vdev is added, target this number of metaslabs per top-level vdev.
346.
347.It Sy zfs_vdev_default_ms_shift Ns = Ns Sy 29 Po 512 MiB Pc Pq int
348Default limit for metaslab size.
349.
350.It Sy zfs_vdev_max_auto_ashift Ns = Ns Sy 14 Pq ulong
351Maximum ashift used when optimizing for logical \[->] physical sector size on new
352top-level vdevs.
353May be increased up to
354.Sy ASHIFT_MAX Po 16 Pc ,
355but this may negatively impact pool space efficiency.
356.
357.It Sy zfs_vdev_min_auto_ashift Ns = Ns Sy ASHIFT_MIN Po 9 Pc Pq ulong
358Minimum ashift used when creating new top-level vdevs.
359.
360.It Sy zfs_vdev_min_ms_count Ns = Ns Sy 16 Pq int
361Minimum number of metaslabs to create in a top-level vdev.
362.
363.It Sy vdev_validate_skip Ns = Ns Sy 0 Ns | Ns 1 Pq int
364Skip label validation steps during pool import.
365Changing is not recommended unless you know what you're doing
366and are recovering a damaged label.
367.
368.It Sy zfs_vdev_ms_count_limit Ns = Ns Sy 131072 Po 128k Pc Pq int
369Practical upper limit of total metaslabs per top-level vdev.
370.
371.It Sy metaslab_preload_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
372Enable metaslab group preloading.
373.
374.It Sy metaslab_lba_weighting_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
375Give more weight to metaslabs with lower LBAs,
376assuming they have greater bandwidth,
377as is typically the case on a modern constant angular velocity disk drive.
378.
379.It Sy metaslab_unload_delay Ns = Ns Sy 32 Pq int
380After a metaslab is used, we keep it loaded for this many TXGs, to attempt to
381reduce unnecessary reloading.
382Note that both this many TXGs and
383.Sy metaslab_unload_delay_ms
384milliseconds must pass before unloading will occur.
385.
386.It Sy metaslab_unload_delay_ms Ns = Ns Sy 600000 Ns ms Po 10 min Pc Pq int
387After a metaslab is used, we keep it loaded for this many milliseconds,
388to attempt to reduce unnecessary reloading.
389Note, that both this many milliseconds and
390.Sy metaslab_unload_delay
391TXGs must pass before unloading will occur.
392.
393.It Sy reference_history Ns = Ns Sy 3 Pq int
394Maximum reference holders being tracked when reference_tracking_enable is active.
395.
396.It Sy reference_tracking_enable Ns = Ns Sy 0 Ns | Ns 1 Pq int
397Track reference holders to
398.Sy refcount_t
399objects (debug builds only).
400.
401.It Sy send_holes_without_birth_time Ns = Ns Sy 1 Ns | Ns 0 Pq int
402When set, the
403.Sy hole_birth
404optimization will not be used, and all holes will always be sent during a
405.Nm zfs Cm send .
406This is useful if you suspect your datasets are affected by a bug in
407.Sy hole_birth .
408.
409.It Sy spa_config_path Ns = Ns Pa /etc/zfs/zpool.cache Pq charp
410SPA config file.
411.
412.It Sy spa_asize_inflation Ns = Ns Sy 24 Pq int
413Multiplication factor used to estimate actual disk consumption from the
414size of data being written.
415The default value is a worst case estimate,
416but lower values may be valid for a given pool depending on its configuration.
417Pool administrators who understand the factors involved
418may wish to specify a more realistic inflation factor,
419particularly if they operate close to quota or capacity limits.
420.
421.It Sy spa_load_print_vdev_tree Ns = Ns Sy 0 Ns | Ns 1 Pq int
422Whether to print the vdev tree in the debugging message buffer during pool import.
423.
424.It Sy spa_load_verify_data Ns = Ns Sy 1 Ns | Ns 0 Pq int
425Whether to traverse data blocks during an "extreme rewind"
426.Pq Fl X
427import.
428.Pp
429An extreme rewind import normally performs a full traversal of all
430blocks in the pool for verification.
431If this parameter is unset, the traversal skips non-metadata blocks.
432It can be toggled once the
433import has started to stop or start the traversal of non-metadata blocks.
434.
435.It Sy spa_load_verify_metadata  Ns = Ns Sy 1 Ns | Ns 0 Pq int
436Whether to traverse blocks during an "extreme rewind"
437.Pq Fl X
438pool import.
439.Pp
440An extreme rewind import normally performs a full traversal of all
441blocks in the pool for verification.
442If this parameter is unset, the traversal is not performed.
443It can be toggled once the import has started to stop or start the traversal.
444.
445.It Sy spa_load_verify_shift Ns = Ns Sy 4 Po 1/16th Pc Pq int
446Sets the maximum number of bytes to consume during pool import to the log2
447fraction of the target ARC size.
448.
449.It Sy spa_slop_shift Ns = Ns Sy 5 Po 1/32nd Pc Pq int
450Normally, we don't allow the last
451.Sy 3.2% Pq Sy 1/2^spa_slop_shift
452of space in the pool to be consumed.
453This ensures that we don't run the pool completely out of space,
454due to unaccounted changes (e.g. to the MOS).
455It also limits the worst-case time to allocate space.
456If we have less than this amount of free space,
457most ZPL operations (e.g. write, create) will return
458.Sy ENOSPC .
459.
460.It Sy spa_upgrade_errlog_limit Ns = Ns Sy 0 Pq uint
461Limits the number of on-disk error log entries that will be converted to the
462new format when enabling the
463.Sy head_errlog
464feature.
465The default is to convert all log entries.
466.
467.It Sy vdev_removal_max_span Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq int
468During top-level vdev removal, chunks of data are copied from the vdev
469which may include free space in order to trade bandwidth for IOPS.
470This parameter determines the maximum span of free space, in bytes,
471which will be included as "unnecessary" data in a chunk of copied data.
472.Pp
473The default value here was chosen to align with
474.Sy zfs_vdev_read_gap_limit ,
475which is a similar concept when doing
476regular reads (but there's no reason it has to be the same).
477.
478.It Sy vdev_file_logical_ashift Ns = Ns Sy 9 Po 512 B Pc Pq ulong
479Logical ashift for file-based devices.
480.
481.It Sy vdev_file_physical_ashift Ns = Ns Sy 9 Po 512 B Pc Pq ulong
482Physical ashift for file-based devices.
483.
484.It Sy zap_iterate_prefetch Ns = Ns Sy 1 Ns | Ns 0 Pq int
485If set, when we start iterating over a ZAP object,
486prefetch the entire object (all leaf blocks).
487However, this is limited by
488.Sy dmu_prefetch_max .
489.
490.It Sy zfetch_array_rd_sz Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq ulong
491If prefetching is enabled, disable prefetching for reads larger than this size.
492.
493.It Sy zfetch_min_distance Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq uint
494Min bytes to prefetch per stream.
495Prefetch distance starts from the demand access size and quickly grows to
496this value, doubling on each hit.
497After that it may grow further by 1/8 per hit, but only if some prefetch
498since last time haven't completed in time to satisfy demand request, i.e.
499prefetch depth didn't cover the read latency or the pool got saturated.
500.
501.It Sy zfetch_max_distance Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq uint
502Max bytes to prefetch per stream.
503.
504.It Sy zfetch_max_idistance Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq uint
505Max bytes to prefetch indirects for per stream.
506.
507.It Sy zfetch_max_streams Ns = Ns Sy 8 Pq uint
508Max number of streams per zfetch (prefetch streams per file).
509.
510.It Sy zfetch_min_sec_reap Ns = Ns Sy 1 Pq uint
511Min time before inactive prefetch stream can be reclaimed
512.
513.It Sy zfetch_max_sec_reap Ns = Ns Sy 2 Pq uint
514Max time before inactive prefetch stream can be deleted
515.
516.It Sy zfs_abd_scatter_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
517Enables ARC from using scatter/gather lists and forces all allocations to be
518linear in kernel memory.
519Disabling can improve performance in some code paths
520at the expense of fragmented kernel memory.
521.
522.It Sy zfs_abd_scatter_max_order Ns = Ns Sy MAX_ORDER\-1 Pq uint
523Maximum number of consecutive memory pages allocated in a single block for
524scatter/gather lists.
525.Pp
526The value of
527.Sy MAX_ORDER
528depends on kernel configuration.
529.
530.It Sy zfs_abd_scatter_min_size Ns = Ns Sy 1536 Ns B Po 1.5 KiB Pc Pq uint
531This is the minimum allocation size that will use scatter (page-based) ABDs.
532Smaller allocations will use linear ABDs.
533.
534.It Sy zfs_arc_dnode_limit Ns = Ns Sy 0 Ns B Pq ulong
535When the number of bytes consumed by dnodes in the ARC exceeds this number of
536bytes, try to unpin some of it in response to demand for non-metadata.
537This value acts as a ceiling to the amount of dnode metadata, and defaults to
538.Sy 0 ,
539which indicates that a percent which is based on
540.Sy zfs_arc_dnode_limit_percent
541of the ARC meta buffers that may be used for dnodes.
542.Pp
543Also see
544.Sy zfs_arc_meta_prune
545which serves a similar purpose but is used
546when the amount of metadata in the ARC exceeds
547.Sy zfs_arc_meta_limit
548rather than in response to overall demand for non-metadata.
549.
550.It Sy zfs_arc_dnode_limit_percent Ns = Ns Sy 10 Ns % Pq ulong
551Percentage that can be consumed by dnodes of ARC meta buffers.
552.Pp
553See also
554.Sy zfs_arc_dnode_limit ,
555which serves a similar purpose but has a higher priority if nonzero.
556.
557.It Sy zfs_arc_dnode_reduce_percent Ns = Ns Sy 10 Ns % Pq ulong
558Percentage of ARC dnodes to try to scan in response to demand for non-metadata
559when the number of bytes consumed by dnodes exceeds
560.Sy zfs_arc_dnode_limit .
561.
562.It Sy zfs_arc_average_blocksize Ns = Ns Sy 8192 Ns B Po 8 KiB Pc Pq int
563The ARC's buffer hash table is sized based on the assumption of an average
564block size of this value.
565This works out to roughly 1 MiB of hash table per 1 GiB of physical memory
566with 8-byte pointers.
567For configurations with a known larger average block size,
568this value can be increased to reduce the memory footprint.
569.
570.It Sy zfs_arc_eviction_pct Ns = Ns Sy 200 Ns % Pq int
571When
572.Fn arc_is_overflowing ,
573.Fn arc_get_data_impl
574waits for this percent of the requested amount of data to be evicted.
575For example, by default, for every
576.Em 2 KiB
577that's evicted,
578.Em 1 KiB
579of it may be "reused" by a new allocation.
580Since this is above
581.Sy 100 Ns % ,
582it ensures that progress is made towards getting
583.Sy arc_size No under Sy arc_c .
584Since this is finite, it ensures that allocations can still happen,
585even during the potentially long time that
586.Sy arc_size No is more than Sy arc_c .
587.
588.It Sy zfs_arc_evict_batch_limit Ns = Ns Sy 10 Pq int
589Number ARC headers to evict per sub-list before proceeding to another sub-list.
590This batch-style operation prevents entire sub-lists from being evicted at once
591but comes at a cost of additional unlocking and locking.
592.
593.It Sy zfs_arc_grow_retry Ns = Ns Sy 0 Ns s Pq int
594If set to a non zero value, it will replace the
595.Sy arc_grow_retry
596value with this value.
597The
598.Sy arc_grow_retry
599.No value Pq default Sy 5 Ns s
600is the number of seconds the ARC will wait before
601trying to resume growth after a memory pressure event.
602.
603.It Sy zfs_arc_lotsfree_percent Ns = Ns Sy 10 Ns % Pq int
604Throttle I/O when free system memory drops below this percentage of total
605system memory.
606Setting this value to
607.Sy 0
608will disable the throttle.
609.
610.It Sy zfs_arc_max Ns = Ns Sy 0 Ns B Pq ulong
611Max size of ARC in bytes.
612If
613.Sy 0 ,
614then the max size of ARC is determined by the amount of system memory installed.
615Under Linux, half of system memory will be used as the limit.
616Under
617.Fx ,
618the larger of
619.Sy all_system_memory No \- Sy 1 GiB
620and
621.Sy 5/8 No \(mu Sy all_system_memory
622will be used as the limit.
623This value must be at least
624.Sy 67108864 Ns B Pq 64 MiB .
625.Pp
626This value can be changed dynamically, with some caveats.
627It cannot be set back to
628.Sy 0
629while running, and reducing it below the current ARC size will not cause
630the ARC to shrink without memory pressure to induce shrinking.
631.
632.It Sy zfs_arc_meta_adjust_restarts Ns = Ns Sy 4096 Pq ulong
633The number of restart passes to make while scanning the ARC attempting
634the free buffers in order to stay below the
635.Sy fs_arc_meta_limit .
636This value should not need to be tuned but is available to facilitate
637performance analysis.
638.
639.It Sy zfs_arc_meta_limit Ns = Ns Sy 0 Ns B Pq ulong
640The maximum allowed size in bytes that metadata buffers are allowed to
641consume in the ARC.
642When this limit is reached, metadata buffers will be reclaimed,
643even if the overall
644.Sy arc_c_max
645has not been reached.
646It defaults to
647.Sy 0 ,
648which indicates that a percentage based on
649.Sy zfs_arc_meta_limit_percent
650of the ARC may be used for metadata.
651.Pp
652This value my be changed dynamically, except that must be set to an explicit value
653.Pq cannot be set back to Sy 0 .
654.
655.It Sy zfs_arc_meta_limit_percent Ns = Ns Sy 75 Ns % Pq ulong
656Percentage of ARC buffers that can be used for metadata.
657.Pp
658See also
659.Sy zfs_arc_meta_limit ,
660which serves a similar purpose but has a higher priority if nonzero.
661.
662.It Sy zfs_arc_meta_min Ns = Ns Sy 0 Ns B Pq ulong
663The minimum allowed size in bytes that metadata buffers may consume in
664the ARC.
665.
666.It Sy zfs_arc_meta_prune Ns = Ns Sy 10000 Pq int
667The number of dentries and inodes to be scanned looking for entries
668which can be dropped.
669This may be required when the ARC reaches the
670.Sy zfs_arc_meta_limit
671because dentries and inodes can pin buffers in the ARC.
672Increasing this value will cause to dentry and inode caches
673to be pruned more aggressively.
674Setting this value to
675.Sy 0
676will disable pruning the inode and dentry caches.
677.
678.It Sy zfs_arc_meta_strategy Ns = Ns Sy 1 Ns | Ns 0 Pq int
679Define the strategy for ARC metadata buffer eviction (meta reclaim strategy):
680.Bl -tag -compact -offset 4n -width "0 (META_ONLY)"
681.It Sy 0 Pq META_ONLY
682evict only the ARC metadata buffers
683.It Sy 1 Pq BALANCED
684additional data buffers may be evicted if required
685to evict the required number of metadata buffers.
686.El
687.
688.It Sy zfs_arc_min Ns = Ns Sy 0 Ns B Pq ulong
689Min size of ARC in bytes.
690.No If set to Sy 0 , arc_c_min
691will default to consuming the larger of
692.Sy 32 MiB
693and
694.Sy all_system_memory No / Sy 32 .
695.
696.It Sy zfs_arc_min_prefetch_ms Ns = Ns Sy 0 Ns ms Ns Po Ns ≡ Ns 1s Pc Pq int
697Minimum time prefetched blocks are locked in the ARC.
698.
699.It Sy zfs_arc_min_prescient_prefetch_ms Ns = Ns Sy 0 Ns ms Ns Po Ns ≡ Ns 6s Pc Pq int
700Minimum time "prescient prefetched" blocks are locked in the ARC.
701These blocks are meant to be prefetched fairly aggressively ahead of
702the code that may use them.
703.
704.It Sy zfs_arc_prune_task_threads Ns = Ns Sy 1 Pq int
705Number of arc_prune threads.
706.Fx
707does not need more than one.
708Linux may theoretically use one per mount point up to number of CPUs,
709but that was not proven to be useful.
710.
711.It Sy zfs_max_missing_tvds Ns = Ns Sy 0 Pq int
712Number of missing top-level vdevs which will be allowed during
713pool import (only in read-only mode).
714.
715.It Sy zfs_max_nvlist_src_size Ns = Sy 0 Pq ulong
716Maximum size in bytes allowed to be passed as
717.Sy zc_nvlist_src_size
718for ioctls on
719.Pa /dev/zfs .
720This prevents a user from causing the kernel to allocate
721an excessive amount of memory.
722When the limit is exceeded, the ioctl fails with
723.Sy EINVAL
724and a description of the error is sent to the
725.Pa zfs-dbgmsg
726log.
727This parameter should not need to be touched under normal circumstances.
728If
729.Sy 0 ,
730equivalent to a quarter of the user-wired memory limit under
731.Fx
732and to
733.Sy 134217728 Ns B Pq 128 MiB
734under Linux.
735.
736.It Sy zfs_multilist_num_sublists Ns = Ns Sy 0 Pq int
737To allow more fine-grained locking, each ARC state contains a series
738of lists for both data and metadata objects.
739Locking is performed at the level of these "sub-lists".
740This parameters controls the number of sub-lists per ARC state,
741and also applies to other uses of the multilist data structure.
742.Pp
743If
744.Sy 0 ,
745equivalent to the greater of the number of online CPUs and
746.Sy 4 .
747.
748.It Sy zfs_arc_overflow_shift Ns = Ns Sy 8 Pq int
749The ARC size is considered to be overflowing if it exceeds the current
750ARC target size
751.Pq Sy arc_c
752by thresholds determined by this parameter.
753Exceeding by
754.Sy ( arc_c No >> Sy zfs_arc_overflow_shift ) No / Sy 2
755starts ARC reclamation process.
756If that appears insufficient, exceeding by
757.Sy ( arc_c No >> Sy zfs_arc_overflow_shift ) No \(mu Sy 1.5
758blocks new buffer allocation until the reclaim thread catches up.
759Started reclamation process continues till ARC size returns below the
760target size.
761.Pp
762The default value of
763.Sy 8
764causes the ARC to start reclamation if it exceeds the target size by
765.Em 0.2%
766of the target size, and block allocations by
767.Em 0.6% .
768.
769.It Sy zfs_arc_p_min_shift Ns = Ns Sy 0 Pq int
770If nonzero, this will update
771.Sy arc_p_min_shift Pq default Sy 4
772with the new value.
773.Sy arc_p_min_shift No is used as a shift of Sy arc_c
774when calculating the minumum
775.Sy arc_p No size.
776.
777.It Sy zfs_arc_p_dampener_disable Ns = Ns Sy 1 Ns | Ns 0 Pq int
778Disable
779.Sy arc_p
780adapt dampener, which reduces the maximum single adjustment to
781.Sy arc_p .
782.
783.It Sy zfs_arc_shrink_shift Ns = Ns Sy 0 Pq int
784If nonzero, this will update
785.Sy arc_shrink_shift Pq default Sy 7
786with the new value.
787.
788.It Sy zfs_arc_pc_percent Ns = Ns Sy 0 Ns % Po off Pc Pq uint
789Percent of pagecache to reclaim ARC to.
790.Pp
791This tunable allows the ZFS ARC to play more nicely
792with the kernel's LRU pagecache.
793It can guarantee that the ARC size won't collapse under scanning
794pressure on the pagecache, yet still allows the ARC to be reclaimed down to
795.Sy zfs_arc_min
796if necessary.
797This value is specified as percent of pagecache size (as measured by
798.Sy NR_FILE_PAGES ) ,
799where that percent may exceed
800.Sy 100 .
801This
802only operates during memory pressure/reclaim.
803.
804.It Sy zfs_arc_shrinker_limit Ns = Ns Sy 10000 Pq int
805This is a limit on how many pages the ARC shrinker makes available for
806eviction in response to one page allocation attempt.
807Note that in practice, the kernel's shrinker can ask us to evict
808up to about four times this for one allocation attempt.
809.Pp
810The default limit of
811.Sy 10000 Pq in practice, Em 160 MiB No per allocation attempt with 4 KiB pages
812limits the amount of time spent attempting to reclaim ARC memory to
813less than 100 ms per allocation attempt,
814even with a small average compressed block size of ~8 KiB.
815.Pp
816The parameter can be set to 0 (zero) to disable the limit,
817and only applies on Linux.
818.
819.It Sy zfs_arc_sys_free Ns = Ns Sy 0 Ns B Pq ulong
820The target number of bytes the ARC should leave as free memory on the system.
821If zero, equivalent to the bigger of
822.Sy 512 KiB No and Sy all_system_memory/64 .
823.
824.It Sy zfs_autoimport_disable Ns = Ns Sy 1 Ns | Ns 0 Pq int
825Disable pool import at module load by ignoring the cache file
826.Pq Sy spa_config_path .
827.
828.It Sy zfs_checksum_events_per_second Ns = Ns Sy 20 Ns /s Pq uint
829Rate limit checksum events to this many per second.
830Note that this should not be set below the ZED thresholds
831(currently 10 checksums over 10 seconds)
832or else the daemon may not trigger any action.
833.
834.It Sy zfs_commit_timeout_pct Ns = Ns Sy 5 Ns % Pq int
835This controls the amount of time that a ZIL block (lwb) will remain "open"
836when it isn't "full", and it has a thread waiting for it to be committed to
837stable storage.
838The timeout is scaled based on a percentage of the last lwb
839latency to avoid significantly impacting the latency of each individual
840transaction record (itx).
841.
842.It Sy zfs_condense_indirect_commit_entry_delay_ms Ns = Ns Sy 0 Ns ms Pq int
843Vdev indirection layer (used for device removal) sleeps for this many
844milliseconds during mapping generation.
845Intended for use with the test suite to throttle vdev removal speed.
846.
847.It Sy zfs_condense_indirect_obsolete_pct Ns = Ns Sy 25 Ns % Pq int
848Minimum percent of obsolete bytes in vdev mapping required to attempt to condense
849.Pq see Sy zfs_condense_indirect_vdevs_enable .
850Intended for use with the test suite
851to facilitate triggering condensing as needed.
852.
853.It Sy zfs_condense_indirect_vdevs_enable Ns = Ns Sy 1 Ns | Ns 0 Pq int
854Enable condensing indirect vdev mappings.
855When set, attempt to condense indirect vdev mappings
856if the mapping uses more than
857.Sy zfs_condense_min_mapping_bytes
858bytes of memory and if the obsolete space map object uses more than
859.Sy zfs_condense_max_obsolete_bytes
860bytes on-disk.
861The condensing process is an attempt to save memory by removing obsolete mappings.
862.
863.It Sy zfs_condense_max_obsolete_bytes Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq ulong
864Only attempt to condense indirect vdev mappings if the on-disk size
865of the obsolete space map object is greater than this number of bytes
866.Pq see Sy zfs_condense_indirect_vdevs_enable .
867.
868.It Sy zfs_condense_min_mapping_bytes Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq ulong
869Minimum size vdev mapping to attempt to condense
870.Pq see Sy zfs_condense_indirect_vdevs_enable .
871.
872.It Sy zfs_dbgmsg_enable Ns = Ns Sy 1 Ns | Ns 0 Pq int
873Internally ZFS keeps a small log to facilitate debugging.
874The log is enabled by default, and can be disabled by unsetting this option.
875The contents of the log can be accessed by reading
876.Pa /proc/spl/kstat/zfs/dbgmsg .
877Writing
878.Sy 0
879to the file clears the log.
880.Pp
881This setting does not influence debug prints due to
882.Sy zfs_flags .
883.
884.It Sy zfs_dbgmsg_maxsize Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq int
885Maximum size of the internal ZFS debug log.
886.
887.It Sy zfs_dbuf_state_index Ns = Ns Sy 0 Pq int
888Historically used for controlling what reporting was available under
889.Pa /proc/spl/kstat/zfs .
890No effect.
891.
892.It Sy zfs_deadman_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
893When a pool sync operation takes longer than
894.Sy zfs_deadman_synctime_ms ,
895or when an individual I/O operation takes longer than
896.Sy zfs_deadman_ziotime_ms ,
897then the operation is considered to be "hung".
898If
899.Sy zfs_deadman_enabled
900is set, then the deadman behavior is invoked as described by
901.Sy zfs_deadman_failmode .
902By default, the deadman is enabled and set to
903.Sy wait
904which results in "hung" I/O operations only being logged.
905The deadman is automatically disabled when a pool gets suspended.
906.
907.It Sy zfs_deadman_failmode Ns = Ns Sy wait Pq charp
908Controls the failure behavior when the deadman detects a "hung" I/O operation.
909Valid values are:
910.Bl -tag -compact -offset 4n -width "continue"
911.It Sy wait
912Wait for a "hung" operation to complete.
913For each "hung" operation a "deadman" event will be posted
914describing that operation.
915.It Sy continue
916Attempt to recover from a "hung" operation by re-dispatching it
917to the I/O pipeline if possible.
918.It Sy panic
919Panic the system.
920This can be used to facilitate automatic fail-over
921to a properly configured fail-over partner.
922.El
923.
924.It Sy zfs_deadman_checktime_ms Ns = Ns Sy 60000 Ns ms Po 1 min Pc Pq int
925Check time in milliseconds.
926This defines the frequency at which we check for hung I/O requests
927and potentially invoke the
928.Sy zfs_deadman_failmode
929behavior.
930.
931.It Sy zfs_deadman_synctime_ms Ns = Ns Sy 600000 Ns ms Po 10 min Pc Pq ulong
932Interval in milliseconds after which the deadman is triggered and also
933the interval after which a pool sync operation is considered to be "hung".
934Once this limit is exceeded the deadman will be invoked every
935.Sy zfs_deadman_checktime_ms
936milliseconds until the pool sync completes.
937.
938.It Sy zfs_deadman_ziotime_ms Ns = Ns Sy 300000 Ns ms Po 5 min Pc Pq ulong
939Interval in milliseconds after which the deadman is triggered and an
940individual I/O operation is considered to be "hung".
941As long as the operation remains "hung",
942the deadman will be invoked every
943.Sy zfs_deadman_checktime_ms
944milliseconds until the operation completes.
945.
946.It Sy zfs_dedup_prefetch Ns = Ns Sy 0 Ns | Ns 1 Pq int
947Enable prefetching dedup-ed blocks which are going to be freed.
948.
949.It Sy zfs_delay_min_dirty_percent Ns = Ns Sy 60 Ns % Pq int
950Start to delay each transaction once there is this amount of dirty data,
951expressed as a percentage of
952.Sy zfs_dirty_data_max .
953This value should be at least
954.Sy zfs_vdev_async_write_active_max_dirty_percent .
955.No See Sx ZFS TRANSACTION DELAY .
956.
957.It Sy zfs_delay_scale Ns = Ns Sy 500000 Pq int
958This controls how quickly the transaction delay approaches infinity.
959Larger values cause longer delays for a given amount of dirty data.
960.Pp
961For the smoothest delay, this value should be about 1 billion divided
962by the maximum number of operations per second.
963This will smoothly handle between ten times and a tenth of this number.
964.No See Sx ZFS TRANSACTION DELAY .
965.Pp
966.Sy zfs_delay_scale No \(mu Sy zfs_dirty_data_max Em must No be smaller than Sy 2^64 .
967.
968.It Sy zfs_disable_ivset_guid_check Ns = Ns Sy 0 Ns | Ns 1 Pq int
969Disables requirement for IVset GUIDs to be present and match when doing a raw
970receive of encrypted datasets.
971Intended for users whose pools were created with
972OpenZFS pre-release versions and now have compatibility issues.
973.
974.It Sy zfs_key_max_salt_uses Ns = Ns Sy 400000000 Po 4*10^8 Pc Pq ulong
975Maximum number of uses of a single salt value before generating a new one for
976encrypted datasets.
977The default value is also the maximum.
978.
979.It Sy zfs_object_mutex_size Ns = Ns Sy 64 Pq uint
980Size of the znode hashtable used for holds.
981.Pp
982Due to the need to hold locks on objects that may not exist yet, kernel mutexes
983are not created per-object and instead a hashtable is used where collisions
984will result in objects waiting when there is not actually contention on the
985same object.
986.
987.It Sy zfs_slow_io_events_per_second Ns = Ns Sy 20 Ns /s Pq int
988Rate limit delay and deadman zevents (which report slow I/O operations) to this many per
989second.
990.
991.It Sy zfs_unflushed_max_mem_amt Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq ulong
992Upper-bound limit for unflushed metadata changes to be held by the
993log spacemap in memory, in bytes.
994.
995.It Sy zfs_unflushed_max_mem_ppm Ns = Ns Sy 1000 Ns ppm Po 0.1% Pc Pq ulong
996Part of overall system memory that ZFS allows to be used
997for unflushed metadata changes by the log spacemap, in millionths.
998.
999.It Sy zfs_unflushed_log_block_max Ns = Ns Sy 131072 Po 128k Pc Pq ulong
1000Describes the maximum number of log spacemap blocks allowed for each pool.
1001The default value means that the space in all the log spacemaps
1002can add up to no more than
1003.Sy 131072
1004blocks (which means
1005.Em 16 GiB
1006of logical space before compression and ditto blocks,
1007assuming that blocksize is
1008.Em 128 KiB ) .
1009.Pp
1010This tunable is important because it involves a trade-off between import
1011time after an unclean export and the frequency of flushing metaslabs.
1012The higher this number is, the more log blocks we allow when the pool is
1013active which means that we flush metaslabs less often and thus decrease
1014the number of I/O operations for spacemap updates per TXG.
1015At the same time though, that means that in the event of an unclean export,
1016there will be more log spacemap blocks for us to read, inducing overhead
1017in the import time of the pool.
1018The lower the number, the amount of flushing increases, destroying log
1019blocks quicker as they become obsolete faster, which leaves less blocks
1020to be read during import time after a crash.
1021.Pp
1022Each log spacemap block existing during pool import leads to approximately
1023one extra logical I/O issued.
1024This is the reason why this tunable is exposed in terms of blocks rather
1025than space used.
1026.
1027.It Sy zfs_unflushed_log_block_min Ns = Ns Sy 1000 Pq ulong
1028If the number of metaslabs is small and our incoming rate is high,
1029we could get into a situation that we are flushing all our metaslabs every TXG.
1030Thus we always allow at least this many log blocks.
1031.
1032.It Sy zfs_unflushed_log_block_pct Ns = Ns Sy 400 Ns % Pq ulong
1033Tunable used to determine the number of blocks that can be used for
1034the spacemap log, expressed as a percentage of the total number of
1035unflushed metaslabs in the pool.
1036.
1037.It Sy zfs_unflushed_log_txg_max Ns = Ns Sy 1000 Pq ulong
1038Tunable limiting maximum time in TXGs any metaslab may remain unflushed.
1039It effectively limits maximum number of unflushed per-TXG spacemap logs
1040that need to be read after unclean pool export.
1041.
1042.It Sy zfs_unlink_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq uint
1043When enabled, files will not be asynchronously removed from the list of pending
1044unlinks and the space they consume will be leaked.
1045Once this option has been disabled and the dataset is remounted,
1046the pending unlinks will be processed and the freed space returned to the pool.
1047This option is used by the test suite.
1048.
1049.It Sy zfs_delete_blocks Ns = Ns Sy 20480 Pq ulong
1050This is the used to define a large file for the purposes of deletion.
1051Files containing more than
1052.Sy zfs_delete_blocks
1053will be deleted asynchronously, while smaller files are deleted synchronously.
1054Decreasing this value will reduce the time spent in an
1055.Xr unlink 2
1056system call, at the expense of a longer delay before the freed space is available.
1057.
1058.It Sy zfs_dirty_data_max Ns = Pq int
1059Determines the dirty space limit in bytes.
1060Once this limit is exceeded, new writes are halted until space frees up.
1061This parameter takes precedence over
1062.Sy zfs_dirty_data_max_percent .
1063.No See Sx ZFS TRANSACTION DELAY .
1064.Pp
1065Defaults to
1066.Sy physical_ram/10 ,
1067capped at
1068.Sy zfs_dirty_data_max_max .
1069.
1070.It Sy zfs_dirty_data_max_max Ns = Pq int
1071Maximum allowable value of
1072.Sy zfs_dirty_data_max ,
1073expressed in bytes.
1074This limit is only enforced at module load time, and will be ignored if
1075.Sy zfs_dirty_data_max
1076is later changed.
1077This parameter takes precedence over
1078.Sy zfs_dirty_data_max_max_percent .
1079.No See Sx ZFS TRANSACTION DELAY .
1080.Pp
1081Defaults to
1082.Sy physical_ram/4 ,
1083.
1084.It Sy zfs_dirty_data_max_max_percent Ns = Ns Sy 25 Ns % Pq int
1085Maximum allowable value of
1086.Sy zfs_dirty_data_max ,
1087expressed as a percentage of physical RAM.
1088This limit is only enforced at module load time, and will be ignored if
1089.Sy zfs_dirty_data_max
1090is later changed.
1091The parameter
1092.Sy zfs_dirty_data_max_max
1093takes precedence over this one.
1094.No See Sx ZFS TRANSACTION DELAY .
1095.
1096.It Sy zfs_dirty_data_max_percent Ns = Ns Sy 10 Ns % Pq int
1097Determines the dirty space limit, expressed as a percentage of all memory.
1098Once this limit is exceeded, new writes are halted until space frees up.
1099The parameter
1100.Sy zfs_dirty_data_max
1101takes precedence over this one.
1102.No See Sx ZFS TRANSACTION DELAY .
1103.Pp
1104Subject to
1105.Sy zfs_dirty_data_max_max .
1106.
1107.It Sy zfs_dirty_data_sync_percent Ns = Ns Sy 20 Ns % Pq int
1108Start syncing out a transaction group if there's at least this much dirty data
1109.Pq as a percentage of Sy zfs_dirty_data_max .
1110This should be less than
1111.Sy zfs_vdev_async_write_active_min_dirty_percent .
1112.
1113.It Sy zfs_wrlog_data_max Ns = Pq int
1114The upper limit of write-transaction zil log data size in bytes.
1115Write operations are throttled when approaching the limit until log data is
1116cleared out after transaction group sync.
1117Because of some overhead, it should be set at least 2 times the size of
1118.Sy zfs_dirty_data_max
1119.No to prevent harming normal write throughput.
1120It also should be smaller than the size of the slog device if slog is present.
1121.Pp
1122Defaults to
1123.Sy zfs_dirty_data_max*2
1124.
1125.It Sy zfs_fallocate_reserve_percent Ns = Ns Sy 110 Ns % Pq uint
1126Since ZFS is a copy-on-write filesystem with snapshots, blocks cannot be
1127preallocated for a file in order to guarantee that later writes will not
1128run out of space.
1129Instead,
1130.Xr fallocate 2
1131space preallocation only checks that sufficient space is currently available
1132in the pool or the user's project quota allocation,
1133and then creates a sparse file of the requested size.
1134The requested space is multiplied by
1135.Sy zfs_fallocate_reserve_percent
1136to allow additional space for indirect blocks and other internal metadata.
1137Setting this to
1138.Sy 0
1139disables support for
1140.Xr fallocate 2
1141and causes it to return
1142.Sy EOPNOTSUPP .
1143.
1144.It Sy zfs_fletcher_4_impl Ns = Ns Sy fastest Pq string
1145Select a fletcher 4 implementation.
1146.Pp
1147Supported selectors are:
1148.Sy fastest , scalar , sse2 , ssse3 , avx2 , avx512f , avx512bw ,
1149.No and Sy aarch64_neon .
1150All except
1151.Sy fastest No and Sy scalar
1152require instruction set extensions to be available,
1153and will only appear if ZFS detects that they are present at runtime.
1154If multiple implementations of fletcher 4 are available, the
1155.Sy fastest
1156will be chosen using a micro benchmark.
1157Selecting
1158.Sy scalar
1159results in the original CPU-based calculation being used.
1160Selecting any option other than
1161.Sy fastest No or Sy scalar
1162results in vector instructions
1163from the respective CPU instruction set being used.
1164.
1165.It Sy zfs_blake3_impl Ns = Ns Sy fastest Pq string
1166Select a BLAKE3 implementation.
1167.Pp
1168Supported selectors are:
1169.Sy cycle , fastest , generic , sse2 , sse41 , avx2 , avx512 .
1170All except
1171.Sy cycle , fastest No and Sy generic
1172require instruction set extensions to be available,
1173and will only appear if ZFS detects that they are present at runtime.
1174If multiple implementations of BLAKE3 are available, the
1175.Sy fastest will be chosen using a micro benchmark. You can see the
1176benchmark results by reading this kstat file:
1177.Pa /proc/spl/kstat/zfs/chksum_bench .
1178.
1179.It Sy zfs_free_bpobj_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
1180Enable/disable the processing of the free_bpobj object.
1181.
1182.It Sy zfs_async_block_max_blocks Ns = Ns Sy ULONG_MAX Po unlimited Pc Pq ulong
1183Maximum number of blocks freed in a single TXG.
1184.
1185.It Sy zfs_max_async_dedup_frees Ns = Ns Sy 100000 Po 10^5 Pc Pq ulong
1186Maximum number of dedup blocks freed in a single TXG.
1187.
1188.It Sy zfs_vdev_async_read_max_active Ns = Ns Sy 3 Pq int
1189Maximum asynchronous read I/O operations active to each device.
1190.No See Sx ZFS I/O SCHEDULER .
1191.
1192.It Sy zfs_vdev_async_read_min_active Ns = Ns Sy 1 Pq int
1193Minimum asynchronous read I/O operation active to each device.
1194.No See Sx ZFS I/O SCHEDULER .
1195.
1196.It Sy zfs_vdev_async_write_active_max_dirty_percent Ns = Ns Sy 60 Ns % Pq int
1197When the pool has more than this much dirty data, use
1198.Sy zfs_vdev_async_write_max_active
1199to limit active async writes.
1200If the dirty data is between the minimum and maximum,
1201the active I/O limit is linearly interpolated.
1202.No See Sx ZFS I/O SCHEDULER .
1203.
1204.It Sy zfs_vdev_async_write_active_min_dirty_percent Ns = Ns Sy 30 Ns % Pq int
1205When the pool has less than this much dirty data, use
1206.Sy zfs_vdev_async_write_min_active
1207to limit active async writes.
1208If the dirty data is between the minimum and maximum,
1209the active I/O limit is linearly
1210interpolated.
1211.No See Sx ZFS I/O SCHEDULER .
1212.
1213.It Sy zfs_vdev_async_write_max_active Ns = Ns Sy 30 Pq int
1214Maximum asynchronous write I/O operations active to each device.
1215.No See Sx ZFS I/O SCHEDULER .
1216.
1217.It Sy zfs_vdev_async_write_min_active Ns = Ns Sy 2 Pq int
1218Minimum asynchronous write I/O operations active to each device.
1219.No See Sx ZFS I/O SCHEDULER .
1220.Pp
1221Lower values are associated with better latency on rotational media but poorer
1222resilver performance.
1223The default value of
1224.Sy 2
1225was chosen as a compromise.
1226A value of
1227.Sy 3
1228has been shown to improve resilver performance further at a cost of
1229further increasing latency.
1230.
1231.It Sy zfs_vdev_initializing_max_active Ns = Ns Sy 1 Pq int
1232Maximum initializing I/O operations active to each device.
1233.No See Sx ZFS I/O SCHEDULER .
1234.
1235.It Sy zfs_vdev_initializing_min_active Ns = Ns Sy 1 Pq int
1236Minimum initializing I/O operations active to each device.
1237.No See Sx ZFS I/O SCHEDULER .
1238.
1239.It Sy zfs_vdev_max_active Ns = Ns Sy 1000 Pq int
1240The maximum number of I/O operations active to each device.
1241Ideally, this will be at least the sum of each queue's
1242.Sy max_active .
1243.No See Sx ZFS I/O SCHEDULER .
1244.
1245.It Sy zfs_vdev_rebuild_max_active Ns = Ns Sy 3 Pq int
1246Maximum sequential resilver I/O operations active to each device.
1247.No See Sx ZFS I/O SCHEDULER .
1248.
1249.It Sy zfs_vdev_rebuild_min_active Ns = Ns Sy 1 Pq int
1250Minimum sequential resilver I/O operations active to each device.
1251.No See Sx ZFS I/O SCHEDULER .
1252.
1253.It Sy zfs_vdev_removal_max_active Ns = Ns Sy 2 Pq int
1254Maximum removal I/O operations active to each device.
1255.No See Sx ZFS I/O SCHEDULER .
1256.
1257.It Sy zfs_vdev_removal_min_active Ns = Ns Sy 1 Pq int
1258Minimum removal I/O operations active to each device.
1259.No See Sx ZFS I/O SCHEDULER .
1260.
1261.It Sy zfs_vdev_scrub_max_active Ns = Ns Sy 2 Pq int
1262Maximum scrub I/O operations active to each device.
1263.No See Sx ZFS I/O SCHEDULER .
1264.
1265.It Sy zfs_vdev_scrub_min_active Ns = Ns Sy 1 Pq int
1266Minimum scrub I/O operations active to each device.
1267.No See Sx ZFS I/O SCHEDULER .
1268.
1269.It Sy zfs_vdev_sync_read_max_active Ns = Ns Sy 10 Pq int
1270Maximum synchronous read I/O operations active to each device.
1271.No See Sx ZFS I/O SCHEDULER .
1272.
1273.It Sy zfs_vdev_sync_read_min_active Ns = Ns Sy 10 Pq int
1274Minimum synchronous read I/O operations active to each device.
1275.No See Sx ZFS I/O SCHEDULER .
1276.
1277.It Sy zfs_vdev_sync_write_max_active Ns = Ns Sy 10 Pq int
1278Maximum synchronous write I/O operations active to each device.
1279.No See Sx ZFS I/O SCHEDULER .
1280.
1281.It Sy zfs_vdev_sync_write_min_active Ns = Ns Sy 10 Pq int
1282Minimum synchronous write I/O operations active to each device.
1283.No See Sx ZFS I/O SCHEDULER .
1284.
1285.It Sy zfs_vdev_trim_max_active Ns = Ns Sy 2 Pq int
1286Maximum trim/discard I/O operations active to each device.
1287.No See Sx ZFS I/O SCHEDULER .
1288.
1289.It Sy zfs_vdev_trim_min_active Ns = Ns Sy 1 Pq int
1290Minimum trim/discard I/O operations active to each device.
1291.No See Sx ZFS I/O SCHEDULER .
1292.
1293.It Sy zfs_vdev_nia_delay Ns = Ns Sy 5 Pq int
1294For non-interactive I/O (scrub, resilver, removal, initialize and rebuild),
1295the number of concurrently-active I/O operations is limited to
1296.Sy zfs_*_min_active ,
1297unless the vdev is "idle".
1298When there are no interactive I/O operations active (synchronous or otherwise),
1299and
1300.Sy zfs_vdev_nia_delay
1301operations have completed since the last interactive operation,
1302then the vdev is considered to be "idle",
1303and the number of concurrently-active non-interactive operations is increased to
1304.Sy zfs_*_max_active .
1305.No See Sx ZFS I/O SCHEDULER .
1306.
1307.It Sy zfs_vdev_nia_credit Ns = Ns Sy 5 Pq int
1308Some HDDs tend to prioritize sequential I/O so strongly, that concurrent
1309random I/O latency reaches several seconds.
1310On some HDDs this happens even if sequential I/O operations
1311are submitted one at a time, and so setting
1312.Sy zfs_*_max_active Ns = Sy 1
1313does not help.
1314To prevent non-interactive I/O, like scrub,
1315from monopolizing the device, no more than
1316.Sy zfs_vdev_nia_credit operations can be sent
1317while there are outstanding incomplete interactive operations.
1318This enforced wait ensures the HDD services the interactive I/O
1319within a reasonable amount of time.
1320.No See Sx ZFS I/O SCHEDULER .
1321.
1322.It Sy zfs_vdev_queue_depth_pct Ns = Ns Sy 1000 Ns % Pq int
1323Maximum number of queued allocations per top-level vdev expressed as
1324a percentage of
1325.Sy zfs_vdev_async_write_max_active ,
1326which allows the system to detect devices that are more capable
1327of handling allocations and to allocate more blocks to those devices.
1328This allows for dynamic allocation distribution when devices are imbalanced,
1329as fuller devices will tend to be slower than empty devices.
1330.Pp
1331Also see
1332.Sy zio_dva_throttle_enabled .
1333.
1334.It Sy zfs_expire_snapshot Ns = Ns Sy 300 Ns s Pq int
1335Time before expiring
1336.Pa .zfs/snapshot .
1337.
1338.It Sy zfs_admin_snapshot Ns = Ns Sy 0 Ns | Ns 1 Pq int
1339Allow the creation, removal, or renaming of entries in the
1340.Sy .zfs/snapshot
1341directory to cause the creation, destruction, or renaming of snapshots.
1342When enabled, this functionality works both locally and over NFS exports
1343which have the
1344.Em no_root_squash
1345option set.
1346.
1347.It Sy zfs_flags Ns = Ns Sy 0 Pq int
1348Set additional debugging flags.
1349The following flags may be bitwise-ored together:
1350.TS
1351box;
1352lbz r l l .
1353	Value	Symbolic Name	Description
1354_
1355	1	ZFS_DEBUG_DPRINTF	Enable dprintf entries in the debug log.
1356*	2	ZFS_DEBUG_DBUF_VERIFY	Enable extra dbuf verifications.
1357*	4	ZFS_DEBUG_DNODE_VERIFY	Enable extra dnode verifications.
1358	8	ZFS_DEBUG_SNAPNAMES	Enable snapshot name verification.
1359	16	ZFS_DEBUG_MODIFY	Check for illegally modified ARC buffers.
1360	64	ZFS_DEBUG_ZIO_FREE	Enable verification of block frees.
1361	128	ZFS_DEBUG_HISTOGRAM_VERIFY	Enable extra spacemap histogram verifications.
1362	256	ZFS_DEBUG_METASLAB_VERIFY	Verify space accounting on disk matches in-memory \fBrange_trees\fP.
1363	512	ZFS_DEBUG_SET_ERROR	Enable \fBSET_ERROR\fP and dprintf entries in the debug log.
1364	1024	ZFS_DEBUG_INDIRECT_REMAP	Verify split blocks created by device removal.
1365	2048	ZFS_DEBUG_TRIM	Verify TRIM ranges are always within the allocatable range tree.
1366	4096	ZFS_DEBUG_LOG_SPACEMAP	Verify that the log summary is consistent with the spacemap log
1367			       and enable \fBzfs_dbgmsgs\fP for metaslab loading and flushing.
1368.TE
1369.Sy \& * No Requires debug build.
1370.
1371.It Sy zfs_btree_verify_intensity Ns = Ns Sy 0 Pq uint
1372Enables btree verification.
1373The following settings are culminative:
1374.TS
1375box;
1376lbz r l l .
1377	Value	Description
1378
1379	1	Verify height.
1380	2	Verify pointers from children to parent.
1381	3	Verify element counts.
1382	4	Verify element order. (expensive)
1383*	5	Verify unused memory is poisoned. (expensive)
1384.TE
1385.Sy \& * No Requires debug build.
1386.
1387.It Sy zfs_free_leak_on_eio Ns = Ns Sy 0 Ns | Ns 1 Pq int
1388If destroy encounters an
1389.Sy EIO
1390while reading metadata (e.g. indirect blocks),
1391space referenced by the missing metadata can not be freed.
1392Normally this causes the background destroy to become "stalled",
1393as it is unable to make forward progress.
1394While in this stalled state, all remaining space to free
1395from the error-encountering filesystem is "temporarily leaked".
1396Set this flag to cause it to ignore the
1397.Sy EIO ,
1398permanently leak the space from indirect blocks that can not be read,
1399and continue to free everything else that it can.
1400.Pp
1401The default "stalling" behavior is useful if the storage partially
1402fails (i.e. some but not all I/O operations fail), and then later recovers.
1403In this case, we will be able to continue pool operations while it is
1404partially failed, and when it recovers, we can continue to free the
1405space, with no leaks.
1406Note, however, that this case is actually fairly rare.
1407.Pp
1408Typically pools either
1409.Bl -enum -compact -offset 4n -width "1."
1410.It
1411fail completely (but perhaps temporarily,
1412e.g. due to a top-level vdev going offline), or
1413.It
1414have localized, permanent errors (e.g. disk returns the wrong data
1415due to bit flip or firmware bug).
1416.El
1417In the former case, this setting does not matter because the
1418pool will be suspended and the sync thread will not be able to make
1419forward progress regardless.
1420In the latter, because the error is permanent, the best we can do
1421is leak the minimum amount of space,
1422which is what setting this flag will do.
1423It is therefore reasonable for this flag to normally be set,
1424but we chose the more conservative approach of not setting it,
1425so that there is no possibility of
1426leaking space in the "partial temporary" failure case.
1427.
1428.It Sy zfs_free_min_time_ms Ns = Ns Sy 1000 Ns ms Po 1s Pc Pq int
1429During a
1430.Nm zfs Cm destroy
1431operation using the
1432.Sy async_destroy
1433feature,
1434a minimum of this much time will be spent working on freeing blocks per TXG.
1435.
1436.It Sy zfs_obsolete_min_time_ms Ns = Ns Sy 500 Ns ms Pq int
1437Similar to
1438.Sy zfs_free_min_time_ms ,
1439but for cleanup of old indirection records for removed vdevs.
1440.
1441.It Sy zfs_immediate_write_sz Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq long
1442Largest data block to write to the ZIL.
1443Larger blocks will be treated as if the dataset being written to had the
1444.Sy logbias Ns = Ns Sy throughput
1445property set.
1446.
1447.It Sy zfs_initialize_value Ns = Ns Sy 16045690984833335022 Po 0xDEADBEEFDEADBEEE Pc Pq ulong
1448Pattern written to vdev free space by
1449.Xr zpool-initialize 8 .
1450.
1451.It Sy zfs_initialize_chunk_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq ulong
1452Size of writes used by
1453.Xr zpool-initialize 8 .
1454This option is used by the test suite.
1455.
1456.It Sy zfs_livelist_max_entries Ns = Ns Sy 500000 Po 5*10^5 Pc Pq ulong
1457The threshold size (in block pointers) at which we create a new sub-livelist.
1458Larger sublists are more costly from a memory perspective but the fewer
1459sublists there are, the lower the cost of insertion.
1460.
1461.It Sy zfs_livelist_min_percent_shared Ns = Ns Sy 75 Ns % Pq int
1462If the amount of shared space between a snapshot and its clone drops below
1463this threshold, the clone turns off the livelist and reverts to the old
1464deletion method.
1465This is in place because livelists no long give us a benefit
1466once a clone has been overwritten enough.
1467.
1468.It Sy zfs_livelist_condense_new_alloc Ns = Ns Sy 0 Pq int
1469Incremented each time an extra ALLOC blkptr is added to a livelist entry while
1470it is being condensed.
1471This option is used by the test suite to track race conditions.
1472.
1473.It Sy zfs_livelist_condense_sync_cancel Ns = Ns Sy 0 Pq int
1474Incremented each time livelist condensing is canceled while in
1475.Fn spa_livelist_condense_sync .
1476This option is used by the test suite to track race conditions.
1477.
1478.It Sy zfs_livelist_condense_sync_pause Ns = Ns Sy 0 Ns | Ns 1 Pq int
1479When set, the livelist condense process pauses indefinitely before
1480executing the synctask \(em
1481.Fn spa_livelist_condense_sync .
1482This option is used by the test suite to trigger race conditions.
1483.
1484.It Sy zfs_livelist_condense_zthr_cancel Ns = Ns Sy 0 Pq int
1485Incremented each time livelist condensing is canceled while in
1486.Fn spa_livelist_condense_cb .
1487This option is used by the test suite to track race conditions.
1488.
1489.It Sy zfs_livelist_condense_zthr_pause Ns = Ns Sy 0 Ns | Ns 1 Pq int
1490When set, the livelist condense process pauses indefinitely before
1491executing the open context condensing work in
1492.Fn spa_livelist_condense_cb .
1493This option is used by the test suite to trigger race conditions.
1494.
1495.It Sy zfs_lua_max_instrlimit Ns = Ns Sy 100000000 Po 10^8 Pc Pq ulong
1496The maximum execution time limit that can be set for a ZFS channel program,
1497specified as a number of Lua instructions.
1498.
1499.It Sy zfs_lua_max_memlimit Ns = Ns Sy 104857600 Po 100 MiB Pc Pq ulong
1500The maximum memory limit that can be set for a ZFS channel program, specified
1501in bytes.
1502.
1503.It Sy zfs_max_dataset_nesting Ns = Ns Sy 50 Pq int
1504The maximum depth of nested datasets.
1505This value can be tuned temporarily to
1506fix existing datasets that exceed the predefined limit.
1507.
1508.It Sy zfs_max_log_walking Ns = Ns Sy 5 Pq ulong
1509The number of past TXGs that the flushing algorithm of the log spacemap
1510feature uses to estimate incoming log blocks.
1511.
1512.It Sy zfs_max_logsm_summary_length Ns = Ns Sy 10 Pq ulong
1513Maximum number of rows allowed in the summary of the spacemap log.
1514.
1515.It Sy zfs_max_recordsize Ns = Ns Sy 16777216 Po 16 MiB Pc Pq int
1516We currently support block sizes from
1517.Em 512 Po 512 B Pc No to Em 16777216 Po 16 MiB Pc .
1518The benefits of larger blocks, and thus larger I/O,
1519need to be weighed against the cost of COWing a giant block to modify one byte.
1520Additionally, very large blocks can have an impact on I/O latency,
1521and also potentially on the memory allocator.
1522Therefore, we formerly forbade creating blocks larger than 1M.
1523Larger blocks could be created by changing it,
1524and pools with larger blocks can always be imported and used,
1525regardless of this setting.
1526.
1527.It Sy zfs_allow_redacted_dataset_mount Ns = Ns Sy 0 Ns | Ns 1 Pq int
1528Allow datasets received with redacted send/receive to be mounted.
1529Normally disabled because these datasets may be missing key data.
1530.
1531.It Sy zfs_min_metaslabs_to_flush Ns = Ns Sy 1 Pq ulong
1532Minimum number of metaslabs to flush per dirty TXG.
1533.
1534.It Sy zfs_metaslab_fragmentation_threshold Ns = Ns Sy 70 Ns % Pq int
1535Allow metaslabs to keep their active state as long as their fragmentation
1536percentage is no more than this value.
1537An active metaslab that exceeds this threshold
1538will no longer keep its active status allowing better metaslabs to be selected.
1539.
1540.It Sy zfs_mg_fragmentation_threshold Ns = Ns Sy 95 Ns % Pq int
1541Metaslab groups are considered eligible for allocations if their
1542fragmentation metric (measured as a percentage) is less than or equal to
1543this value.
1544If a metaslab group exceeds this threshold then it will be
1545skipped unless all metaslab groups within the metaslab class have also
1546crossed this threshold.
1547.
1548.It Sy zfs_mg_noalloc_threshold Ns = Ns Sy 0 Ns % Pq int
1549Defines a threshold at which metaslab groups should be eligible for allocations.
1550The value is expressed as a percentage of free space
1551beyond which a metaslab group is always eligible for allocations.
1552If a metaslab group's free space is less than or equal to the
1553threshold, the allocator will avoid allocating to that group
1554unless all groups in the pool have reached the threshold.
1555Once all groups have reached the threshold, all groups are allowed to accept
1556allocations.
1557The default value of
1558.Sy 0
1559disables the feature and causes all metaslab groups to be eligible for allocations.
1560.Pp
1561This parameter allows one to deal with pools having heavily imbalanced
1562vdevs such as would be the case when a new vdev has been added.
1563Setting the threshold to a non-zero percentage will stop allocations
1564from being made to vdevs that aren't filled to the specified percentage
1565and allow lesser filled vdevs to acquire more allocations than they
1566otherwise would under the old
1567.Sy zfs_mg_alloc_failures
1568facility.
1569.
1570.It Sy zfs_ddt_data_is_special Ns = Ns Sy 1 Ns | Ns 0 Pq int
1571If enabled, ZFS will place DDT data into the special allocation class.
1572.
1573.It Sy zfs_user_indirect_is_special Ns = Ns Sy 1 Ns | Ns 0 Pq int
1574If enabled, ZFS will place user data indirect blocks
1575into the special allocation class.
1576.
1577.It Sy zfs_multihost_history Ns = Ns Sy 0 Pq int
1578Historical statistics for this many latest multihost updates will be available in
1579.Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /multihost .
1580.
1581.It Sy zfs_multihost_interval Ns = Ns Sy 1000 Ns ms Po 1 s Pc Pq ulong
1582Used to control the frequency of multihost writes which are performed when the
1583.Sy multihost
1584pool property is on.
1585This is one of the factors used to determine the
1586length of the activity check during import.
1587.Pp
1588The multihost write period is
1589.Sy zfs_multihost_interval No / Sy leaf-vdevs .
1590On average a multihost write will be issued for each leaf vdev
1591every
1592.Sy zfs_multihost_interval
1593milliseconds.
1594In practice, the observed period can vary with the I/O load
1595and this observed value is the delay which is stored in the uberblock.
1596.
1597.It Sy zfs_multihost_import_intervals Ns = Ns Sy 20 Pq uint
1598Used to control the duration of the activity test on import.
1599Smaller values of
1600.Sy zfs_multihost_import_intervals
1601will reduce the import time but increase
1602the risk of failing to detect an active pool.
1603The total activity check time is never allowed to drop below one second.
1604.Pp
1605On import the activity check waits a minimum amount of time determined by
1606.Sy zfs_multihost_interval No \(mu Sy zfs_multihost_import_intervals ,
1607or the same product computed on the host which last had the pool imported,
1608whichever is greater.
1609The activity check time may be further extended if the value of MMP
1610delay found in the best uberblock indicates actual multihost updates happened
1611at longer intervals than
1612.Sy zfs_multihost_interval .
1613A minimum of
1614.Em 100 ms
1615is enforced.
1616.Pp
1617.Sy 0 No is equivalent to Sy 1 .
1618.
1619.It Sy zfs_multihost_fail_intervals Ns = Ns Sy 10 Pq uint
1620Controls the behavior of the pool when multihost write failures or delays are
1621detected.
1622.Pp
1623When
1624.Sy 0 ,
1625multihost write failures or delays are ignored.
1626The failures will still be reported to the ZED which depending on
1627its configuration may take action such as suspending the pool or offlining a
1628device.
1629.Pp
1630Otherwise, the pool will be suspended if
1631.Sy zfs_multihost_fail_intervals No \(mu Sy zfs_multihost_interval
1632milliseconds pass without a successful MMP write.
1633This guarantees the activity test will see MMP writes if the pool is imported.
1634.Sy 1 No is equivalent to Sy 2 ;
1635this is necessary to prevent the pool from being suspended
1636due to normal, small I/O latency variations.
1637.
1638.It Sy zfs_no_scrub_io Ns = Ns Sy 0 Ns | Ns 1 Pq int
1639Set to disable scrub I/O.
1640This results in scrubs not actually scrubbing data and
1641simply doing a metadata crawl of the pool instead.
1642.
1643.It Sy zfs_no_scrub_prefetch Ns = Ns Sy 0 Ns | Ns 1 Pq int
1644Set to disable block prefetching for scrubs.
1645.
1646.It Sy zfs_nocacheflush Ns = Ns Sy 0 Ns | Ns 1 Pq int
1647Disable cache flush operations on disks when writing.
1648Setting this will cause pool corruption on power loss
1649if a volatile out-of-order write cache is enabled.
1650.
1651.It Sy zfs_nopwrite_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
1652Allow no-operation writes.
1653The occurrence of nopwrites will further depend on other pool properties
1654.Pq i.a. the checksumming and compression algorithms .
1655.
1656.It Sy zfs_dmu_offset_next_sync Ns = Ns Sy 1 Ns | Ns 0 Pq int
1657Enable forcing TXG sync to find holes.
1658When enabled forces ZFS to sync data when
1659.Sy SEEK_HOLE No or Sy SEEK_DATA
1660flags are used allowing holes in a file to be accurately reported.
1661When disabled holes will not be reported in recently dirtied files.
1662.
1663.It Sy zfs_pd_bytes_max Ns = Ns Sy 52428800 Ns B Po 50 MiB Pc Pq int
1664The number of bytes which should be prefetched during a pool traversal, like
1665.Nm zfs Cm send
1666or other data crawling operations.
1667.
1668.It Sy zfs_traverse_indirect_prefetch_limit Ns = Ns Sy 32 Pq int
1669The number of blocks pointed by indirect (non-L0) block which should be
1670prefetched during a pool traversal, like
1671.Nm zfs Cm send
1672or other data crawling operations.
1673.
1674.It Sy zfs_per_txg_dirty_frees_percent Ns = Ns Sy 5 Ns % Pq ulong
1675Control percentage of dirtied indirect blocks from frees allowed into one TXG.
1676After this threshold is crossed, additional frees will wait until the next TXG.
1677.Sy 0 No disables this throttle.
1678.
1679.It Sy zfs_prefetch_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int
1680Disable predictive prefetch.
1681Note that it leaves "prescient" prefetch
1682.Pq for, e.g., Nm zfs Cm send
1683intact.
1684Unlike predictive prefetch, prescient prefetch never issues I/O
1685that ends up not being needed, so it can't hurt performance.
1686.
1687.It Sy zfs_qat_checksum_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int
1688Disable QAT hardware acceleration for SHA256 checksums.
1689May be unset after the ZFS modules have been loaded to initialize the QAT
1690hardware as long as support is compiled in and the QAT driver is present.
1691.
1692.It Sy zfs_qat_compress_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int
1693Disable QAT hardware acceleration for gzip compression.
1694May be unset after the ZFS modules have been loaded to initialize the QAT
1695hardware as long as support is compiled in and the QAT driver is present.
1696.
1697.It Sy zfs_qat_encrypt_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int
1698Disable QAT hardware acceleration for AES-GCM encryption.
1699May be unset after the ZFS modules have been loaded to initialize the QAT
1700hardware as long as support is compiled in and the QAT driver is present.
1701.
1702.It Sy zfs_vnops_read_chunk_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq long
1703Bytes to read per chunk.
1704.
1705.It Sy zfs_read_history Ns = Ns Sy 0 Pq int
1706Historical statistics for this many latest reads will be available in
1707.Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /reads .
1708.
1709.It Sy zfs_read_history_hits Ns = Ns Sy 0 Ns | Ns 1 Pq int
1710Include cache hits in read history
1711.
1712.It Sy zfs_rebuild_max_segment Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq ulong
1713Maximum read segment size to issue when sequentially resilvering a
1714top-level vdev.
1715.
1716.It Sy zfs_rebuild_scrub_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
1717Automatically start a pool scrub when the last active sequential resilver
1718completes in order to verify the checksums of all blocks which have been
1719resilvered.
1720This is enabled by default and strongly recommended.
1721.
1722.It Sy zfs_rebuild_vdev_limit Ns = Ns Sy 33554432 Ns B Po 32 MiB Pc Pq ulong
1723Maximum amount of I/O that can be concurrently issued for a sequential
1724resilver per leaf device, given in bytes.
1725.
1726.It Sy zfs_reconstruct_indirect_combinations_max Ns = Ns Sy 4096 Pq int
1727If an indirect split block contains more than this many possible unique
1728combinations when being reconstructed, consider it too computationally
1729expensive to check them all.
1730Instead, try at most this many randomly selected
1731combinations each time the block is accessed.
1732This allows all segment copies to participate fairly
1733in the reconstruction when all combinations
1734cannot be checked and prevents repeated use of one bad copy.
1735.
1736.It Sy zfs_recover Ns = Ns Sy 0 Ns | Ns 1 Pq int
1737Set to attempt to recover from fatal errors.
1738This should only be used as a last resort,
1739as it typically results in leaked space, or worse.
1740.
1741.It Sy zfs_removal_ignore_errors Ns = Ns Sy 0 Ns | Ns 1 Pq int
1742Ignore hard I/O errors during device removal.
1743When set, if a device encounters a hard I/O error during the removal process
1744the removal will not be cancelled.
1745This can result in a normally recoverable block becoming permanently damaged
1746and is hence not recommended.
1747This should only be used as a last resort when the
1748pool cannot be returned to a healthy state prior to removing the device.
1749.
1750.It Sy zfs_removal_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq int
1751This is used by the test suite so that it can ensure that certain actions
1752happen while in the middle of a removal.
1753.
1754.It Sy zfs_remove_max_segment Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq int
1755The largest contiguous segment that we will attempt to allocate when removing
1756a device.
1757If there is a performance problem with attempting to allocate large blocks,
1758consider decreasing this.
1759The default value is also the maximum.
1760.
1761.It Sy zfs_resilver_disable_defer Ns = Ns Sy 0 Ns | Ns 1 Pq int
1762Ignore the
1763.Sy resilver_defer
1764feature, causing an operation that would start a resilver to
1765immediately restart the one in progress.
1766.
1767.It Sy zfs_resilver_min_time_ms Ns = Ns Sy 3000 Ns ms Po 3 s Pc Pq int
1768Resilvers are processed by the sync thread.
1769While resilvering, it will spend at least this much time
1770working on a resilver between TXG flushes.
1771.
1772.It Sy zfs_scan_ignore_errors Ns = Ns Sy 0 Ns | Ns 1 Pq int
1773If set, remove the DTL (dirty time list) upon completion of a pool scan (scrub),
1774even if there were unrepairable errors.
1775Intended to be used during pool repair or recovery to
1776stop resilvering when the pool is next imported.
1777.
1778.It Sy zfs_scrub_min_time_ms Ns = Ns Sy 1000 Ns ms Po 1 s Pc Pq int
1779Scrubs are processed by the sync thread.
1780While scrubbing, it will spend at least this much time
1781working on a scrub between TXG flushes.
1782.
1783.It Sy zfs_scan_checkpoint_intval Ns = Ns Sy 7200 Ns s Po 2 hour Pc Pq int
1784To preserve progress across reboots, the sequential scan algorithm periodically
1785needs to stop metadata scanning and issue all the verification I/O to disk.
1786The frequency of this flushing is determined by this tunable.
1787.
1788.It Sy zfs_scan_fill_weight Ns = Ns Sy 3 Pq int
1789This tunable affects how scrub and resilver I/O segments are ordered.
1790A higher number indicates that we care more about how filled in a segment is,
1791while a lower number indicates we care more about the size of the extent without
1792considering the gaps within a segment.
1793This value is only tunable upon module insertion.
1794Changing the value afterwards will have no effect on scrub or resilver performance.
1795.
1796.It Sy zfs_scan_issue_strategy Ns = Ns Sy 0 Pq int
1797Determines the order that data will be verified while scrubbing or resilvering:
1798.Bl -tag -compact -offset 4n -width "a"
1799.It Sy 1
1800Data will be verified as sequentially as possible, given the
1801amount of memory reserved for scrubbing
1802.Pq see Sy zfs_scan_mem_lim_fact .
1803This may improve scrub performance if the pool's data is very fragmented.
1804.It Sy 2
1805The largest mostly-contiguous chunk of found data will be verified first.
1806By deferring scrubbing of small segments, we may later find adjacent data
1807to coalesce and increase the segment size.
1808.It Sy 0
1809.No Use strategy Sy 1 No during normal verification
1810.No and strategy Sy 2 No while taking a checkpoint.
1811.El
1812.
1813.It Sy zfs_scan_legacy Ns = Ns Sy 0 Ns | Ns 1 Pq int
1814If unset, indicates that scrubs and resilvers will gather metadata in
1815memory before issuing sequential I/O.
1816Otherwise indicates that the legacy algorithm will be used,
1817where I/O is initiated as soon as it is discovered.
1818Unsetting will not affect scrubs or resilvers that are already in progress.
1819.
1820.It Sy zfs_scan_max_ext_gap Ns = Ns Sy 2097152 Ns B Po 2 MiB Pc Pq int
1821Sets the largest gap in bytes between scrub/resilver I/O operations
1822that will still be considered sequential for sorting purposes.
1823Changing this value will not
1824affect scrubs or resilvers that are already in progress.
1825.
1826.It Sy zfs_scan_mem_lim_fact Ns = Ns Sy 20 Ns ^-1 Pq int
1827Maximum fraction of RAM used for I/O sorting by sequential scan algorithm.
1828This tunable determines the hard limit for I/O sorting memory usage.
1829When the hard limit is reached we stop scanning metadata and start issuing
1830data verification I/O.
1831This is done until we get below the soft limit.
1832.
1833.It Sy zfs_scan_mem_lim_soft_fact Ns = Ns Sy 20 Ns ^-1 Pq int
1834The fraction of the hard limit used to determined the soft limit for I/O sorting
1835by the sequential scan algorithm.
1836When we cross this limit from below no action is taken.
1837When we cross this limit from above it is because we are issuing verification I/O.
1838In this case (unless the metadata scan is done) we stop issuing verification I/O
1839and start scanning metadata again until we get to the hard limit.
1840.
1841.It Sy zfs_scan_strict_mem_lim Ns = Ns Sy 0 Ns | Ns 1 Pq int
1842Enforce tight memory limits on pool scans when a sequential scan is in progress.
1843When disabled, the memory limit may be exceeded by fast disks.
1844.
1845.It Sy zfs_scan_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq int
1846Freezes a scrub/resilver in progress without actually pausing it.
1847Intended for testing/debugging.
1848.
1849.It Sy zfs_scan_vdev_limit Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq int
1850Maximum amount of data that can be concurrently issued at once for scrubs and
1851resilvers per leaf device, given in bytes.
1852.
1853.It Sy zfs_send_corrupt_data Ns = Ns Sy 0 Ns | Ns 1 Pq int
1854Allow sending of corrupt data (ignore read/checksum errors when sending).
1855.
1856.It Sy zfs_send_unmodified_spill_blocks Ns = Ns Sy 1 Ns | Ns 0 Pq int
1857Include unmodified spill blocks in the send stream.
1858Under certain circumstances, previous versions of ZFS could incorrectly
1859remove the spill block from an existing object.
1860Including unmodified copies of the spill blocks creates a backwards-compatible
1861stream which will recreate a spill block if it was incorrectly removed.
1862.
1863.It Sy zfs_send_no_prefetch_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq int
1864The fill fraction of the
1865.Nm zfs Cm send
1866internal queues.
1867The fill fraction controls the timing with which internal threads are woken up.
1868.
1869.It Sy zfs_send_no_prefetch_queue_length Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq int
1870The maximum number of bytes allowed in
1871.Nm zfs Cm send Ns 's
1872internal queues.
1873.
1874.It Sy zfs_send_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq int
1875The fill fraction of the
1876.Nm zfs Cm send
1877prefetch queue.
1878The fill fraction controls the timing with which internal threads are woken up.
1879.
1880.It Sy zfs_send_queue_length Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq int
1881The maximum number of bytes allowed that will be prefetched by
1882.Nm zfs Cm send .
1883This value must be at least twice the maximum block size in use.
1884.
1885.It Sy zfs_recv_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq int
1886The fill fraction of the
1887.Nm zfs Cm receive
1888queue.
1889The fill fraction controls the timing with which internal threads are woken up.
1890.
1891.It Sy zfs_recv_queue_length Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq int
1892The maximum number of bytes allowed in the
1893.Nm zfs Cm receive
1894queue.
1895This value must be at least twice the maximum block size in use.
1896.
1897.It Sy zfs_recv_write_batch_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq int
1898The maximum amount of data, in bytes, that
1899.Nm zfs Cm receive
1900will write in one DMU transaction.
1901This is the uncompressed size, even when receiving a compressed send stream.
1902This setting will not reduce the write size below a single block.
1903Capped at a maximum of
1904.Sy 32 MiB .
1905.
1906.It Sy zfs_recv_best_effort_corrective Ns = Ns Sy 0 Pq int
1907When this variable is set to non-zero a corrective receive:
1908.Bl -enum -compact -offset 4n -width "1."
1909.It
1910Does not enforce the restriction of source & destination snapshot GUIDs
1911matching.
1912.It
1913If there is an error during healing, the healing receive is not
1914terminated instead it moves on to the next record.
1915.El
1916.
1917.It Sy zfs_override_estimate_recordsize Ns = Ns Sy 0 Ns | Ns 1 Pq ulong
1918Setting this variable overrides the default logic for estimating block
1919sizes when doing a
1920.Nm zfs Cm send .
1921The default heuristic is that the average block size
1922will be the current recordsize.
1923Override this value if most data in your dataset is not of that size
1924and you require accurate zfs send size estimates.
1925.
1926.It Sy zfs_sync_pass_deferred_free Ns = Ns Sy 2 Pq int
1927Flushing of data to disk is done in passes.
1928Defer frees starting in this pass.
1929.
1930.It Sy zfs_spa_discard_memory_limit Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq int
1931Maximum memory used for prefetching a checkpoint's space map on each
1932vdev while discarding the checkpoint.
1933.
1934.It Sy zfs_special_class_metadata_reserve_pct Ns = Ns Sy 25 Ns % Pq int
1935Only allow small data blocks to be allocated on the special and dedup vdev
1936types when the available free space percentage on these vdevs exceeds this value.
1937This ensures reserved space is available for pool metadata as the
1938special vdevs approach capacity.
1939.
1940.It Sy zfs_sync_pass_dont_compress Ns = Ns Sy 8 Pq int
1941Starting in this sync pass, disable compression (including of metadata).
1942With the default setting, in practice, we don't have this many sync passes,
1943so this has no effect.
1944.Pp
1945The original intent was that disabling compression would help the sync passes
1946to converge.
1947However, in practice, disabling compression increases
1948the average number of sync passes; because when we turn compression off,
1949many blocks' size will change, and thus we have to re-allocate
1950(not overwrite) them.
1951It also increases the number of
1952.Em 128 KiB
1953allocations (e.g. for indirect blocks and spacemaps)
1954because these will not be compressed.
1955The
1956.Em 128 KiB
1957allocations are especially detrimental to performance
1958on highly fragmented systems, which may have very few free segments of this size,
1959and may need to load new metaslabs to satisfy these allocations.
1960.
1961.It Sy zfs_sync_pass_rewrite Ns = Ns Sy 2 Pq int
1962Rewrite new block pointers starting in this pass.
1963.
1964.It Sy zfs_sync_taskq_batch_pct Ns = Ns Sy 75 Ns % Pq int
1965This controls the number of threads used by
1966.Sy dp_sync_taskq .
1967The default value of
1968.Sy 75%
1969will create a maximum of one thread per CPU.
1970.
1971.It Sy zfs_trim_extent_bytes_max Ns = Ns Sy 134217728 Ns B Po 128 MiB Pc Pq uint
1972Maximum size of TRIM command.
1973Larger ranges will be split into chunks no larger than this value before issuing.
1974.
1975.It Sy zfs_trim_extent_bytes_min Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint
1976Minimum size of TRIM commands.
1977TRIM ranges smaller than this will be skipped,
1978unless they're part of a larger range which was chunked.
1979This is done because it's common for these small TRIMs
1980to negatively impact overall performance.
1981.
1982.It Sy zfs_trim_metaslab_skip Ns = Ns Sy 0 Ns | Ns 1 Pq uint
1983Skip uninitialized metaslabs during the TRIM process.
1984This option is useful for pools constructed from large thinly-provisioned devices
1985where TRIM operations are slow.
1986As a pool ages, an increasing fraction of the pool's metaslabs
1987will be initialized, progressively degrading the usefulness of this option.
1988This setting is stored when starting a manual TRIM and will
1989persist for the duration of the requested TRIM.
1990.
1991.It Sy zfs_trim_queue_limit Ns = Ns Sy 10 Pq uint
1992Maximum number of queued TRIMs outstanding per leaf vdev.
1993The number of concurrent TRIM commands issued to the device is controlled by
1994.Sy zfs_vdev_trim_min_active No and Sy zfs_vdev_trim_max_active .
1995.
1996.It Sy zfs_trim_txg_batch Ns = Ns Sy 32 Pq uint
1997The number of transaction groups' worth of frees which should be aggregated
1998before TRIM operations are issued to the device.
1999This setting represents a trade-off between issuing larger,
2000more efficient TRIM operations and the delay
2001before the recently trimmed space is available for use by the device.
2002.Pp
2003Increasing this value will allow frees to be aggregated for a longer time.
2004This will result is larger TRIM operations and potentially increased memory usage.
2005Decreasing this value will have the opposite effect.
2006The default of
2007.Sy 32
2008was determined to be a reasonable compromise.
2009.
2010.It Sy zfs_txg_history Ns = Ns Sy 0 Pq int
2011Historical statistics for this many latest TXGs will be available in
2012.Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /TXGs .
2013.
2014.It Sy zfs_txg_timeout Ns = Ns Sy 5 Ns s Pq int
2015Flush dirty data to disk at least every this many seconds (maximum TXG duration).
2016.
2017.It Sy zfs_vdev_aggregate_trim Ns = Ns Sy 0 Ns | Ns 1 Pq int
2018Allow TRIM I/O operations to be aggregated.
2019This is normally not helpful because the extents to be trimmed
2020will have been already been aggregated by the metaslab.
2021This option is provided for debugging and performance analysis.
2022.
2023.It Sy zfs_vdev_aggregation_limit Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq int
2024Max vdev I/O aggregation size.
2025.
2026.It Sy zfs_vdev_aggregation_limit_non_rotating Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq int
2027Max vdev I/O aggregation size for non-rotating media.
2028.
2029.It Sy zfs_vdev_cache_bshift Ns = Ns Sy 16 Po 64 KiB Pc Pq int
2030Shift size to inflate reads to.
2031.
2032.It Sy zfs_vdev_cache_max Ns = Ns Sy 16384 Ns B Po 16 KiB Pc Pq int
2033Inflate reads smaller than this value to meet the
2034.Sy zfs_vdev_cache_bshift
2035size
2036.Pq default Sy 64 KiB .
2037.
2038.It Sy zfs_vdev_cache_size Ns = Ns Sy 0 Pq int
2039Total size of the per-disk cache in bytes.
2040.Pp
2041Currently this feature is disabled, as it has been found to not be helpful
2042for performance and in some cases harmful.
2043.
2044.It Sy zfs_vdev_mirror_rotating_inc Ns = Ns Sy 0 Pq int
2045A number by which the balancing algorithm increments the load calculation for
2046the purpose of selecting the least busy mirror member when an I/O operation
2047immediately follows its predecessor on rotational vdevs
2048for the purpose of making decisions based on load.
2049.
2050.It Sy zfs_vdev_mirror_rotating_seek_inc Ns = Ns Sy 5 Pq int
2051A number by which the balancing algorithm increments the load calculation for
2052the purpose of selecting the least busy mirror member when an I/O operation
2053lacks locality as defined by
2054.Sy zfs_vdev_mirror_rotating_seek_offset .
2055Operations within this that are not immediately following the previous operation
2056are incremented by half.
2057.
2058.It Sy zfs_vdev_mirror_rotating_seek_offset Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq int
2059The maximum distance for the last queued I/O operation in which
2060the balancing algorithm considers an operation to have locality.
2061.No See Sx ZFS I/O SCHEDULER .
2062.
2063.It Sy zfs_vdev_mirror_non_rotating_inc Ns = Ns Sy 0 Pq int
2064A number by which the balancing algorithm increments the load calculation for
2065the purpose of selecting the least busy mirror member on non-rotational vdevs
2066when I/O operations do not immediately follow one another.
2067.
2068.It Sy zfs_vdev_mirror_non_rotating_seek_inc Ns = Ns Sy 1 Pq int
2069A number by which the balancing algorithm increments the load calculation for
2070the purpose of selecting the least busy mirror member when an I/O operation lacks
2071locality as defined by the
2072.Sy zfs_vdev_mirror_rotating_seek_offset .
2073Operations within this that are not immediately following the previous operation
2074are incremented by half.
2075.
2076.It Sy zfs_vdev_read_gap_limit Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq int
2077Aggregate read I/O operations if the on-disk gap between them is within this
2078threshold.
2079.
2080.It Sy zfs_vdev_write_gap_limit Ns = Ns Sy 4096 Ns B Po 4 KiB Pc Pq int
2081Aggregate write I/O operations if the on-disk gap between them is within this
2082threshold.
2083.
2084.It Sy zfs_vdev_raidz_impl Ns = Ns Sy fastest Pq string
2085Select the raidz parity implementation to use.
2086.Pp
2087Variants that don't depend on CPU-specific features
2088may be selected on module load, as they are supported on all systems.
2089The remaining options may only be set after the module is loaded,
2090as they are available only if the implementations are compiled in
2091and supported on the running system.
2092.Pp
2093Once the module is loaded,
2094.Pa /sys/module/zfs/parameters/zfs_vdev_raidz_impl
2095will show the available options,
2096with the currently selected one enclosed in square brackets.
2097.Pp
2098.TS
2099lb l l .
2100fastest	selected by built-in benchmark
2101original	original implementation
2102scalar	scalar implementation
2103sse2	SSE2 instruction set	64-bit x86
2104ssse3	SSSE3 instruction set	64-bit x86
2105avx2	AVX2 instruction set	64-bit x86
2106avx512f	AVX512F instruction set	64-bit x86
2107avx512bw	AVX512F & AVX512BW instruction sets	64-bit x86
2108aarch64_neon	NEON	Aarch64/64-bit ARMv8
2109aarch64_neonx2	NEON with more unrolling	Aarch64/64-bit ARMv8
2110powerpc_altivec	Altivec	PowerPC
2111.TE
2112.
2113.It Sy zfs_vdev_scheduler Pq charp
2114.Sy DEPRECATED .
2115Prints warning to kernel log for compatibility.
2116.
2117.It Sy zfs_zevent_len_max Ns = Ns Sy 512 Pq int
2118Max event queue length.
2119Events in the queue can be viewed with
2120.Xr zpool-events 8 .
2121.
2122.It Sy zfs_zevent_retain_max Ns = Ns Sy 2000 Pq int
2123Maximum recent zevent records to retain for duplicate checking.
2124Setting this to
2125.Sy 0
2126disables duplicate detection.
2127.
2128.It Sy zfs_zevent_retain_expire_secs Ns = Ns Sy 900 Ns s Po 15 min Pc Pq int
2129Lifespan for a recent ereport that was retained for duplicate checking.
2130.
2131.It Sy zfs_zil_clean_taskq_maxalloc Ns = Ns Sy 1048576 Pq int
2132The maximum number of taskq entries that are allowed to be cached.
2133When this limit is exceeded transaction records (itxs)
2134will be cleaned synchronously.
2135.
2136.It Sy zfs_zil_clean_taskq_minalloc Ns = Ns Sy 1024 Pq int
2137The number of taskq entries that are pre-populated when the taskq is first
2138created and are immediately available for use.
2139.
2140.It Sy zfs_zil_clean_taskq_nthr_pct Ns = Ns Sy 100 Ns % Pq int
2141This controls the number of threads used by
2142.Sy dp_zil_clean_taskq .
2143The default value of
2144.Sy 100%
2145will create a maximum of one thread per cpu.
2146.
2147.It Sy zil_maxblocksize Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq int
2148This sets the maximum block size used by the ZIL.
2149On very fragmented pools, lowering this
2150.Pq typically to Sy 36 KiB
2151can improve performance.
2152.
2153.It Sy zil_nocacheflush Ns = Ns Sy 0 Ns | Ns 1 Pq int
2154Disable the cache flush commands that are normally sent to disk by
2155the ZIL after an LWB write has completed.
2156Setting this will cause ZIL corruption on power loss
2157if a volatile out-of-order write cache is enabled.
2158.
2159.It Sy zil_replay_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int
2160Disable intent logging replay.
2161Can be disabled for recovery from corrupted ZIL.
2162.
2163.It Sy zil_slog_bulk Ns = Ns Sy 786432 Ns B Po 768 KiB Pc Pq ulong
2164Limit SLOG write size per commit executed with synchronous priority.
2165Any writes above that will be executed with lower (asynchronous) priority
2166to limit potential SLOG device abuse by single active ZIL writer.
2167.
2168.It Sy zfs_zil_saxattr Ns = Ns Sy 1 Ns | Ns 0 Pq int
2169Setting this tunable to zero disables ZIL logging of new
2170.Sy xattr Ns = Ns Sy sa
2171records if the
2172.Sy org.openzfs:zilsaxattr
2173feature is enabled on the pool.
2174This would only be necessary to work around bugs in the ZIL logging or replay
2175code for this record type.
2176The tunable has no effect if the feature is disabled.
2177.
2178.It Sy zfs_embedded_slog_min_ms Ns = Ns Sy 64 Pq int
2179Usually, one metaslab from each normal-class vdev is dedicated for use by
2180the ZIL to log synchronous writes.
2181However, if there are fewer than
2182.Sy zfs_embedded_slog_min_ms
2183metaslabs in the vdev, this functionality is disabled.
2184This ensures that we don't set aside an unreasonable amount of space for the ZIL.
2185.
2186.It Sy zstd_earlyabort_pass Ns = Ns Sy 1 Pq int
2187Whether heuristic for detection of incompressible data with zstd levels >= 3
2188using LZ4 and zstd-1 passes is enabled.
2189.
2190.It Sy zstd_abort_size Ns = Ns Sy 131072 Pq int
2191Minimal uncompressed size (inclusive) of a record before the early abort
2192heuristic will be attempted.
2193.
2194.It Sy zio_deadman_log_all Ns = Ns Sy 0 Ns | Ns 1 Pq int
2195If non-zero, the zio deadman will produce debugging messages
2196.Pq see Sy zfs_dbgmsg_enable
2197for all zios, rather than only for leaf zios possessing a vdev.
2198This is meant to be used by developers to gain
2199diagnostic information for hang conditions which don't involve a mutex
2200or other locking primitive: typically conditions in which a thread in
2201the zio pipeline is looping indefinitely.
2202.
2203.It Sy zio_slow_io_ms Ns = Ns Sy 30000 Ns ms Po 30 s Pc Pq int
2204When an I/O operation takes more than this much time to complete,
2205it's marked as slow.
2206Each slow operation causes a delay zevent.
2207Slow I/O counters can be seen with
2208.Nm zpool Cm status Fl s .
2209.
2210.It Sy zio_dva_throttle_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int
2211Throttle block allocations in the I/O pipeline.
2212This allows for dynamic allocation distribution when devices are imbalanced.
2213When enabled, the maximum number of pending allocations per top-level vdev
2214is limited by
2215.Sy zfs_vdev_queue_depth_pct .
2216.
2217.It Sy zfs_xattr_compat Ns = Ns 0 Ns | Ns 1 Pq int
2218Control the naming scheme used when setting new xattrs in the user namespace.
2219If
2220.Sy 0
2221.Pq the default on Linux ,
2222user namespace xattr names are prefixed with the namespace, to be backwards
2223compatible with previous versions of ZFS on Linux.
2224If
2225.Sy 1
2226.Pq the default on Fx ,
2227user namespace xattr names are not prefixed, to be backwards compatible with
2228previous versions of ZFS on illumos and
2229.Fx .
2230.Pp
2231Either naming scheme can be read on this and future versions of ZFS, regardless
2232of this tunable, but legacy ZFS on illumos or
2233.Fx
2234are unable to read user namespace xattrs written in the Linux format, and
2235legacy versions of ZFS on Linux are unable to read user namespace xattrs written
2236in the legacy ZFS format.
2237.Pp
2238An existing xattr with the alternate naming scheme is removed when overwriting
2239the xattr so as to not accumulate duplicates.
2240.
2241.It Sy zio_requeue_io_start_cut_in_line Ns = Ns Sy 0 Ns | Ns 1 Pq int
2242Prioritize requeued I/O.
2243.
2244.It Sy zio_taskq_batch_pct Ns = Ns Sy 80 Ns % Pq uint
2245Percentage of online CPUs which will run a worker thread for I/O.
2246These workers are responsible for I/O work such as compression and
2247checksum calculations.
2248Fractional number of CPUs will be rounded down.
2249.Pp
2250The default value of
2251.Sy 80%
2252was chosen to avoid using all CPUs which can result in
2253latency issues and inconsistent application performance,
2254especially when slower compression and/or checksumming is enabled.
2255.
2256.It Sy zio_taskq_batch_tpq Ns = Ns Sy 0 Pq uint
2257Number of worker threads per taskq.
2258Lower values improve I/O ordering and CPU utilization,
2259while higher reduces lock contention.
2260.Pp
2261If
2262.Sy 0 ,
2263generate a system-dependent value close to 6 threads per taskq.
2264.
2265.It Sy zvol_inhibit_dev Ns = Ns Sy 0 Ns | Ns 1 Pq uint
2266Do not create zvol device nodes.
2267This may slightly improve startup time on
2268systems with a very large number of zvols.
2269.
2270.It Sy zvol_major Ns = Ns Sy 230 Pq uint
2271Major number for zvol block devices.
2272.
2273.It Sy zvol_max_discard_blocks Ns = Ns Sy 16384 Pq ulong
2274Discard (TRIM) operations done on zvols will be done in batches of this
2275many blocks, where block size is determined by the
2276.Sy volblocksize
2277property of a zvol.
2278.
2279.It Sy zvol_prefetch_bytes Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint
2280When adding a zvol to the system, prefetch this many bytes
2281from the start and end of the volume.
2282Prefetching these regions of the volume is desirable,
2283because they are likely to be accessed immediately by
2284.Xr blkid 8
2285or the kernel partitioner.
2286.
2287.It Sy zvol_request_sync Ns = Ns Sy 0 Ns | Ns 1 Pq uint
2288When processing I/O requests for a zvol, submit them synchronously.
2289This effectively limits the queue depth to
2290.Em 1
2291for each I/O submitter.
2292When unset, requests are handled asynchronously by a thread pool.
2293The number of requests which can be handled concurrently is controlled by
2294.Sy zvol_threads .
2295.Sy zvol_request_sync
2296is ignored when running on a kernel that supports block multiqueue
2297.Pq Li blk-mq .
2298.
2299.It Sy zvol_threads Ns = Ns Sy 0 Pq uint
2300The number of system wide threads to use for processing zvol block IOs.
2301If
2302.Sy 0
2303(the default) then internally set
2304.Sy zvol_threads
2305to the number of CPUs present or 32 (whichever is greater).
2306.
2307.It Sy zvol_blk_mq_threads Ns = Ns Sy 0 Pq uint
2308The number of threads per zvol to use for queuing IO requests.
2309This parameter will only appear if your kernel supports
2310.Li blk-mq
2311and is only read and assigned to a zvol at zvol load time.
2312If
2313.Sy 0
2314(the default) then internally set
2315.Sy zvol_blk_mq_threads
2316to the number of CPUs present.
2317.
2318.It Sy zvol_use_blk_mq Ns = Ns Sy 0 Ns | Ns 1 Pq uint
2319Set to
2320.Sy 1
2321to use the
2322.Li blk-mq
2323API for zvols.
2324Set to
2325.Sy 0
2326(the default) to use the legacy zvol APIs.
2327This setting can give better or worse zvol performance depending on
2328the workload.
2329This parameter will only appear if your kernel supports
2330.Li blk-mq
2331and is only read and assigned to a zvol at zvol load time.
2332.
2333.It Sy zvol_blk_mq_blocks_per_thread Ns = Ns Sy 8 Pq uint
2334If
2335.Sy zvol_use_blk_mq
2336is enabled, then process this number of
2337.Sy volblocksize Ns -sized blocks per zvol thread.
2338This tunable can be use to favor better performance for zvol reads (lower
2339values) or writes (higher values).
2340If set to
2341.Sy 0 ,
2342then the zvol layer will process the maximum number of blocks
2343per thread that it can.
2344This parameter will only appear if your kernel supports
2345.Li blk-mq
2346and is only applied at each zvol's load time.
2347.
2348.It Sy zvol_blk_mq_queue_depth Ns = Ns Sy 0 Pq uint
2349The queue_depth value for the zvol
2350.Li blk-mq
2351interface.
2352This parameter will only appear if your kernel supports
2353.Li blk-mq
2354and is only applied at each zvol's load time.
2355If
2356.Sy 0
2357(the default) then use the kernel's default queue depth.
2358Values are clamped to the kernel's
2359.Dv BLKDEV_MIN_RQ
2360and
2361.Dv BLKDEV_MAX_RQ Ns / Ns Dv BLKDEV_DEFAULT_RQ
2362limits.
2363.
2364.It Sy zvol_volmode Ns = Ns Sy 1 Pq uint
2365Defines zvol block devices behaviour when
2366.Sy volmode Ns = Ns Sy default :
2367.Bl -tag -compact -offset 4n -width "a"
2368.It Sy 1
2369.No equivalent to Sy full
2370.It Sy 2
2371.No equivalent to Sy dev
2372.It Sy 3
2373.No equivalent to Sy none
2374.El
2375.El
2376.
2377.Sh ZFS I/O SCHEDULER
2378ZFS issues I/O operations to leaf vdevs to satisfy and complete I/O operations.
2379The scheduler determines when and in what order those operations are issued.
2380The scheduler divides operations into five I/O classes,
2381prioritized in the following order: sync read, sync write, async read,
2382async write, and scrub/resilver.
2383Each queue defines the minimum and maximum number of concurrent operations
2384that may be issued to the device.
2385In addition, the device has an aggregate maximum,
2386.Sy zfs_vdev_max_active .
2387Note that the sum of the per-queue minima must not exceed the aggregate maximum.
2388If the sum of the per-queue maxima exceeds the aggregate maximum,
2389then the number of active operations may reach
2390.Sy zfs_vdev_max_active ,
2391in which case no further operations will be issued,
2392regardless of whether all per-queue minima have been met.
2393.Pp
2394For many physical devices, throughput increases with the number of
2395concurrent operations, but latency typically suffers.
2396Furthermore, physical devices typically have a limit
2397at which more concurrent operations have no
2398effect on throughput or can actually cause it to decrease.
2399.Pp
2400The scheduler selects the next operation to issue by first looking for an
2401I/O class whose minimum has not been satisfied.
2402Once all are satisfied and the aggregate maximum has not been hit,
2403the scheduler looks for classes whose maximum has not been satisfied.
2404Iteration through the I/O classes is done in the order specified above.
2405No further operations are issued
2406if the aggregate maximum number of concurrent operations has been hit,
2407or if there are no operations queued for an I/O class that has not hit its maximum.
2408Every time an I/O operation is queued or an operation completes,
2409the scheduler looks for new operations to issue.
2410.Pp
2411In general, smaller
2412.Sy max_active Ns s
2413will lead to lower latency of synchronous operations.
2414Larger
2415.Sy max_active Ns s
2416may lead to higher overall throughput, depending on underlying storage.
2417.Pp
2418The ratio of the queues'
2419.Sy max_active Ns s
2420determines the balance of performance between reads, writes, and scrubs.
2421For example, increasing
2422.Sy zfs_vdev_scrub_max_active
2423will cause the scrub or resilver to complete more quickly,
2424but reads and writes to have higher latency and lower throughput.
2425.Pp
2426All I/O classes have a fixed maximum number of outstanding operations,
2427except for the async write class.
2428Asynchronous writes represent the data that is committed to stable storage
2429during the syncing stage for transaction groups.
2430Transaction groups enter the syncing state periodically,
2431so the number of queued async writes will quickly burst up
2432and then bleed down to zero.
2433Rather than servicing them as quickly as possible,
2434the I/O scheduler changes the maximum number of active async write operations
2435according to the amount of dirty data in the pool.
2436Since both throughput and latency typically increase with the number of
2437concurrent operations issued to physical devices, reducing the
2438burstiness in the number of concurrent operations also stabilizes the
2439response time of operations from other – and in particular synchronous – queues.
2440In broad strokes, the I/O scheduler will issue more concurrent operations
2441from the async write queue as there's more dirty data in the pool.
2442.
2443.Ss Async Writes
2444The number of concurrent operations issued for the async write I/O class
2445follows a piece-wise linear function defined by a few adjustable points:
2446.Bd -literal
2447       |              o---------| <-- \fBzfs_vdev_async_write_max_active\fP
2448  ^    |             /^         |
2449  |    |            / |         |
2450active |           /  |         |
2451 I/O   |          /   |         |
2452count  |         /    |         |
2453       |        /     |         |
2454       |-------o      |         | <-- \fBzfs_vdev_async_write_min_active\fP
2455      0|_______^______|_________|
2456       0%      |      |       100% of \fBzfs_dirty_data_max\fP
2457               |      |
2458               |      `-- \fBzfs_vdev_async_write_active_max_dirty_percent\fP
2459               `--------- \fBzfs_vdev_async_write_active_min_dirty_percent\fP
2460.Ed
2461.Pp
2462Until the amount of dirty data exceeds a minimum percentage of the dirty
2463data allowed in the pool, the I/O scheduler will limit the number of
2464concurrent operations to the minimum.
2465As that threshold is crossed, the number of concurrent operations issued
2466increases linearly to the maximum at the specified maximum percentage
2467of the dirty data allowed in the pool.
2468.Pp
2469Ideally, the amount of dirty data on a busy pool will stay in the sloped
2470part of the function between
2471.Sy zfs_vdev_async_write_active_min_dirty_percent
2472and
2473.Sy zfs_vdev_async_write_active_max_dirty_percent .
2474If it exceeds the maximum percentage,
2475this indicates that the rate of incoming data is
2476greater than the rate that the backend storage can handle.
2477In this case, we must further throttle incoming writes,
2478as described in the next section.
2479.
2480.Sh ZFS TRANSACTION DELAY
2481We delay transactions when we've determined that the backend storage
2482isn't able to accommodate the rate of incoming writes.
2483.Pp
2484If there is already a transaction waiting, we delay relative to when
2485that transaction will finish waiting.
2486This way the calculated delay time
2487is independent of the number of threads concurrently executing transactions.
2488.Pp
2489If we are the only waiter, wait relative to when the transaction started,
2490rather than the current time.
2491This credits the transaction for "time already served",
2492e.g. reading indirect blocks.
2493.Pp
2494The minimum time for a transaction to take is calculated as
2495.D1 min_time = min( Ns Sy zfs_delay_scale No \(mu Po Sy dirty No \- Sy min Pc / Po Sy max No \- Sy dirty Pc , 100ms)
2496.Pp
2497The delay has two degrees of freedom that can be adjusted via tunables.
2498The percentage of dirty data at which we start to delay is defined by
2499.Sy zfs_delay_min_dirty_percent .
2500This should typically be at or above
2501.Sy zfs_vdev_async_write_active_max_dirty_percent ,
2502so that we only start to delay after writing at full speed
2503has failed to keep up with the incoming write rate.
2504The scale of the curve is defined by
2505.Sy zfs_delay_scale .
2506Roughly speaking, this variable determines the amount of delay at the midpoint of the curve.
2507.Bd -literal
2508delay
2509 10ms +-------------------------------------------------------------*+
2510      |                                                             *|
2511  9ms +                                                             *+
2512      |                                                             *|
2513  8ms +                                                             *+
2514      |                                                            * |
2515  7ms +                                                            * +
2516      |                                                            * |
2517  6ms +                                                            * +
2518      |                                                            * |
2519  5ms +                                                           *  +
2520      |                                                           *  |
2521  4ms +                                                           *  +
2522      |                                                           *  |
2523  3ms +                                                          *   +
2524      |                                                          *   |
2525  2ms +                                              (midpoint) *    +
2526      |                                                  |    **     |
2527  1ms +                                                  v ***       +
2528      |             \fBzfs_delay_scale\fP ---------->     ********         |
2529    0 +-------------------------------------*********----------------+
2530      0%                    <- \fBzfs_dirty_data_max\fP ->               100%
2531.Ed
2532.Pp
2533Note, that since the delay is added to the outstanding time remaining on the
2534most recent transaction it's effectively the inverse of IOPS.
2535Here, the midpoint of
2536.Em 500 us
2537translates to
2538.Em 2000 IOPS .
2539The shape of the curve
2540was chosen such that small changes in the amount of accumulated dirty data
2541in the first three quarters of the curve yield relatively small differences
2542in the amount of delay.
2543.Pp
2544The effects can be easier to understand when the amount of delay is
2545represented on a logarithmic scale:
2546.Bd -literal
2547delay
2548100ms +-------------------------------------------------------------++
2549      +                                                              +
2550      |                                                              |
2551      +                                                             *+
2552 10ms +                                                             *+
2553      +                                                           ** +
2554      |                                              (midpoint)  **  |
2555      +                                                  |     **    +
2556  1ms +                                                  v ****      +
2557      +             \fBzfs_delay_scale\fP ---------->        *****         +
2558      |                                             ****             |
2559      +                                          ****                +
2560100us +                                        **                    +
2561      +                                       *                      +
2562      |                                      *                       |
2563      +                                     *                        +
2564 10us +                                     *                        +
2565      +                                                              +
2566      |                                                              |
2567      +                                                              +
2568      +--------------------------------------------------------------+
2569      0%                    <- \fBzfs_dirty_data_max\fP ->               100%
2570.Ed
2571.Pp
2572Note here that only as the amount of dirty data approaches its limit does
2573the delay start to increase rapidly.
2574The goal of a properly tuned system should be to keep the amount of dirty data
2575out of that range by first ensuring that the appropriate limits are set
2576for the I/O scheduler to reach optimal throughput on the back-end storage,
2577and then by changing the value of
2578.Sy zfs_delay_scale
2579to increase the steepness of the curve.
2580