1.\" 2.\" The contents of this file are subject to the terms of the Common Development 3.\" and Distribution License (the "License"). You may not use this file except 4.\" in compliance with the License. You can obtain a copy of the license at 5.\" usr/src/OPENSOLARIS.LICENSE or https://opensource.org/licenses/CDDL-1.0. 6.\" 7.\" See the License for the specific language governing permissions and 8.\" limitations under the License. When distributing Covered Code, include this 9.\" CDDL HEADER in each file and include the License file at 10.\" usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this 11.\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your 12.\" own identifying information: 13.\" Portions Copyright [yyyy] [name of copyright owner] 14.\" 15.\" Copyright 2013 Turbo Fredriksson <turbo@bayour.com>. All rights reserved. 16.\" 17.Dd August 24, 2020 18.Dt SPL 4 19.Os 20. 21.Sh NAME 22.Nm spl 23.Nd parameters of the SPL kernel module 24. 25.Sh DESCRIPTION 26.Bl -tag -width Ds 27.It Sy spl_kmem_cache_kmem_threads Ns = Ns Sy 4 Pq uint 28The number of threads created for the spl_kmem_cache task queue. 29This task queue is responsible for allocating new slabs 30for use by the kmem caches. 31For the majority of systems and workloads only a small number of threads are 32required. 33. 34.It Sy spl_kmem_cache_reclaim Ns = Ns Sy 0 Pq uint 35When this is set it prevents Linux from being able to rapidly reclaim all the 36memory held by the kmem caches. 37This may be useful in circumstances where it's preferable that Linux 38reclaim memory from some other subsystem first. 39Setting this will increase the likelihood out of memory events on a memory 40constrained system. 41. 42.It Sy spl_kmem_cache_obj_per_slab Ns = Ns Sy 8 Pq uint 43The preferred number of objects per slab in the cache. 44In general, a larger value will increase the caches memory footprint 45while decreasing the time required to perform an allocation. 46Conversely, a smaller value will minimize the footprint 47and improve cache reclaim time but individual allocations may take longer. 48. 49.It Sy spl_kmem_cache_max_size Ns = Ns Sy 32 Po 64-bit Pc or Sy 4 Po 32-bit Pc Pq uint 50The maximum size of a kmem cache slab in MiB. 51This effectively limits the maximum cache object size to 52.Sy spl_kmem_cache_max_size Ns / Ns Sy spl_kmem_cache_obj_per_slab . 53.Pp 54Caches may not be created with 55object sized larger than this limit. 56. 57.It Sy spl_kmem_cache_slab_limit Ns = Ns Sy 16384 Pq uint 58For small objects the Linux slab allocator should be used to make the most 59efficient use of the memory. 60However, large objects are not supported by 61the Linux slab and therefore the SPL implementation is preferred. 62This value is used to determine the cutoff between a small and large object. 63.Pp 64Objects of size 65.Sy spl_kmem_cache_slab_limit 66or smaller will be allocated using the Linux slab allocator, 67large objects use the SPL allocator. 68A cutoff of 16K was determined to be optimal for architectures using 4K pages. 69. 70.It Sy spl_kmem_alloc_warn Ns = Ns Sy 32768 Pq uint 71As a general rule 72.Fn kmem_alloc 73allocations should be small, 74preferably just a few pages, since they must by physically contiguous. 75Therefore, a rate limited warning will be printed to the console for any 76.Fn kmem_alloc 77which exceeds a reasonable threshold. 78.Pp 79The default warning threshold is set to eight pages but capped at 32K to 80accommodate systems using large pages. 81This value was selected to be small enough to ensure 82the largest allocations are quickly noticed and fixed. 83But large enough to avoid logging any warnings when a allocation size is 84larger than optimal but not a serious concern. 85Since this value is tunable, developers are encouraged to set it lower 86when testing so any new largish allocations are quickly caught. 87These warnings may be disabled by setting the threshold to zero. 88. 89.It Sy spl_kmem_alloc_max Ns = Ns Sy KMALLOC_MAX_SIZE Ns / Ns Sy 4 Pq uint 90Large 91.Fn kmem_alloc 92allocations will fail if they exceed 93.Sy KMALLOC_MAX_SIZE . 94Allocations which are marginally smaller than this limit may succeed but 95should still be avoided due to the expense of locating a contiguous range 96of free pages. 97Therefore, a maximum kmem size with reasonable safely margin of 4x is set. 98.Fn kmem_alloc 99allocations larger than this maximum will quickly fail. 100.Fn vmem_alloc 101allocations less than or equal to this value will use 102.Fn kmalloc , 103but shift to 104.Fn vmalloc 105when exceeding this value. 106. 107.It Sy spl_kmem_cache_magazine_size Ns = Ns Sy 0 Pq uint 108Cache magazines are an optimization designed to minimize the cost of 109allocating memory. 110They do this by keeping a per-cpu cache of recently 111freed objects, which can then be reallocated without taking a lock. 112This can improve performance on highly contended caches. 113However, because objects in magazines will prevent otherwise empty slabs 114from being immediately released this may not be ideal for low memory machines. 115.Pp 116For this reason, 117.Sy spl_kmem_cache_magazine_size 118can be used to set a maximum magazine size. 119When this value is set to 0 the magazine size will 120be automatically determined based on the object size. 121Otherwise magazines will be limited to 2-256 objects per magazine (i.e per cpu). 122Magazines may never be entirely disabled in this implementation. 123. 124.It Sy spl_hostid Ns = Ns Sy 0 Pq ulong 125The system hostid, when set this can be used to uniquely identify a system. 126By default this value is set to zero which indicates the hostid is disabled. 127It can be explicitly enabled by placing a unique non-zero value in 128.Pa /etc/hostid . 129. 130.It Sy spl_hostid_path Ns = Ns Pa /etc/hostid Pq charp 131The expected path to locate the system hostid when specified. 132This value may be overridden for non-standard configurations. 133. 134.It Sy spl_panic_halt Ns = Ns Sy 0 Pq uint 135Cause a kernel panic on assertion failures. 136When not enabled, the thread is halted to facilitate further debugging. 137.Pp 138Set to a non-zero value to enable. 139. 140.It Sy spl_taskq_kick Ns = Ns Sy 0 Pq uint 141Kick stuck taskq to spawn threads. 142When writing a non-zero value to it, it will scan all the taskqs. 143If any of them have a pending task more than 5 seconds old, 144it will kick it to spawn more threads. 145This can be used if you find a rare 146deadlock occurs because one or more taskqs didn't spawn a thread when it should. 147. 148.It Sy spl_taskq_thread_bind Ns = Ns Sy 0 Pq int 149Bind taskq threads to specific CPUs. 150When enabled all taskq threads will be distributed evenly 151across the available CPUs. 152By default, this behavior is disabled to allow the Linux scheduler 153the maximum flexibility to determine where a thread should run. 154. 155.It Sy spl_taskq_thread_dynamic Ns = Ns Sy 1 Pq int 156Allow dynamic taskqs. 157When enabled taskqs which set the 158.Sy TASKQ_DYNAMIC 159flag will by default create only a single thread. 160New threads will be created on demand up to a maximum allowed number 161to facilitate the completion of outstanding tasks. 162Threads which are no longer needed will be promptly destroyed. 163By default this behavior is enabled but it can be disabled to 164aid performance analysis or troubleshooting. 165. 166.It Sy spl_taskq_thread_priority Ns = Ns Sy 1 Pq int 167Allow newly created taskq threads to set a non-default scheduler priority. 168When enabled, the priority specified when a taskq is created will be applied 169to all threads created by that taskq. 170When disabled all threads will use the default Linux kernel thread priority. 171By default, this behavior is enabled. 172. 173.It Sy spl_taskq_thread_sequential Ns = Ns Sy 4 Pq int 174The number of items a taskq worker thread must handle without interruption 175before requesting a new worker thread be spawned. 176This is used to control 177how quickly taskqs ramp up the number of threads processing the queue. 178Because Linux thread creation and destruction are relatively inexpensive a 179small default value has been selected. 180This means that normally threads will be created aggressively which is 181desirable. 182Increasing this value will 183result in a slower thread creation rate which may be preferable for some 184configurations. 185. 186.It Sy spl_max_show_tasks Ns = Ns Sy 512 Pq uint 187The maximum number of tasks per pending list in each taskq shown in 188.Pa /proc/spl/taskq{,-all} . 189Write 190.Sy 0 191to turn off the limit. 192The proc file will walk the lists with lock held, 193reading it could cause a lock-up if the list grow too large 194without limiting the output. 195"(truncated)" will be shown if the list is larger than the limit. 196. 197.It Sy spl_taskq_thread_timeout_ms Ns = Ns Sy 10000 Pq uint 198(Linux-only) 199How long a taskq has to have had no work before we tear it down. 200Previously, we would tear down a dynamic taskq worker as soon 201as we noticed it had no work, but it was observed that this led 202to a lot of churn in tearing down things we then immediately 203spawned anew. 204In practice, it seems any nonzero value will remove the vast 205majority of this churn, while the nontrivially larger value 206was chosen to help filter out the little remaining churn on 207a mostly idle system. 208Setting this value to 209.Sy 0 210will revert to the previous behavior. 211.El 212