1.. _slub: 2 3========================== 4Short users guide for SLUB 5========================== 6 7The basic philosophy of SLUB is very different from SLAB. SLAB 8requires rebuilding the kernel to activate debug options for all 9slab caches. SLUB always includes full debugging but it is off by default. 10SLUB can enable debugging only for selected slabs in order to avoid 11an impact on overall system performance which may make a bug more 12difficult to find. 13 14In order to switch debugging on one can add an option ``slub_debug`` 15to the kernel command line. That will enable full debugging for 16all slabs. 17 18Typically one would then use the ``slabinfo`` command to get statistical 19data and perform operation on the slabs. By default ``slabinfo`` only lists 20slabs that have data in them. See "slabinfo -h" for more options when 21running the command. ``slabinfo`` can be compiled with 22:: 23 24 gcc -o slabinfo tools/vm/slabinfo.c 25 26Some of the modes of operation of ``slabinfo`` require that slub debugging 27be enabled on the command line. F.e. no tracking information will be 28available without debugging on and validation can only partially 29be performed if debugging was not switched on. 30 31Some more sophisticated uses of slub_debug: 32------------------------------------------- 33 34Parameters may be given to ``slub_debug``. If none is specified then full 35debugging is enabled. Format: 36 37slub_debug=<Debug-Options> 38 Enable options for all slabs 39 40slub_debug=<Debug-Options>,<slab name1>,<slab name2>,... 41 Enable options only for select slabs (no spaces 42 after a comma) 43 44Multiple blocks of options for all slabs or selected slabs can be given, with 45blocks of options delimited by ';'. The last of "all slabs" blocks is applied 46to all slabs except those that match one of the "select slabs" block. Options 47of the first "select slabs" blocks that matches the slab's name are applied. 48 49Possible debug options are:: 50 51 F Sanity checks on (enables SLAB_DEBUG_CONSISTENCY_CHECKS 52 Sorry SLAB legacy issues) 53 Z Red zoning 54 P Poisoning (object and padding) 55 U User tracking (free and alloc) 56 T Trace (please only use on single slabs) 57 A Enable failslab filter mark for the cache 58 O Switch debugging off for caches that would have 59 caused higher minimum slab orders 60 - Switch all debugging off (useful if the kernel is 61 configured with CONFIG_SLUB_DEBUG_ON) 62 63F.e. in order to boot just with sanity checks and red zoning one would specify:: 64 65 slub_debug=FZ 66 67Trying to find an issue in the dentry cache? Try:: 68 69 slub_debug=,dentry 70 71to only enable debugging on the dentry cache. You may use an asterisk at the 72end of the slab name, in order to cover all slabs with the same prefix. For 73example, here's how you can poison the dentry cache as well as all kmalloc 74slabs:: 75 76 slub_debug=P,kmalloc-*,dentry 77 78Red zoning and tracking may realign the slab. We can just apply sanity checks 79to the dentry cache with:: 80 81 slub_debug=F,dentry 82 83Debugging options may require the minimum possible slab order to increase as 84a result of storing the metadata (for example, caches with PAGE_SIZE object 85sizes). This has a higher liklihood of resulting in slab allocation errors 86in low memory situations or if there's high fragmentation of memory. To 87switch off debugging for such caches by default, use:: 88 89 slub_debug=O 90 91You can apply different options to different list of slab names, using blocks 92of options. This will enable red zoning for dentry and user tracking for 93kmalloc. All other slabs will not get any debugging enabled:: 94 95 slub_debug=Z,dentry;U,kmalloc-* 96 97You can also enable options (e.g. sanity checks and poisoning) for all caches 98except some that are deemed too performance critical and don't need to be 99debugged by specifying global debug options followed by a list of slab names 100with "-" as options:: 101 102 slub_debug=FZ;-,zs_handle,zspage 103 104The state of each debug option for a slab can be found in the respective files 105under:: 106 107 /sys/kernel/slab/<slab name>/ 108 109If the file contains 1, the option is enabled, 0 means disabled. The debug 110options from the ``slub_debug`` parameter translate to the following files:: 111 112 F sanity_checks 113 Z red_zone 114 P poison 115 U store_user 116 T trace 117 A failslab 118 119Careful with tracing: It may spew out lots of information and never stop if 120used on the wrong slab. 121 122Slab merging 123============ 124 125If no debug options are specified then SLUB may merge similar slabs together 126in order to reduce overhead and increase cache hotness of objects. 127``slabinfo -a`` displays which slabs were merged together. 128 129Slab validation 130=============== 131 132SLUB can validate all object if the kernel was booted with slub_debug. In 133order to do so you must have the ``slabinfo`` tool. Then you can do 134:: 135 136 slabinfo -v 137 138which will test all objects. Output will be generated to the syslog. 139 140This also works in a more limited way if boot was without slab debug. 141In that case ``slabinfo -v`` simply tests all reachable objects. Usually 142these are in the cpu slabs and the partial slabs. Full slabs are not 143tracked by SLUB in a non debug situation. 144 145Getting more performance 146======================== 147 148To some degree SLUB's performance is limited by the need to take the 149list_lock once in a while to deal with partial slabs. That overhead is 150governed by the order of the allocation for each slab. The allocations 151can be influenced by kernel parameters: 152 153.. slub_min_objects=x (default 4) 154.. slub_min_order=x (default 0) 155.. slub_max_order=x (default 3 (PAGE_ALLOC_COSTLY_ORDER)) 156 157``slub_min_objects`` 158 allows to specify how many objects must at least fit into one 159 slab in order for the allocation order to be acceptable. In 160 general slub will be able to perform this number of 161 allocations on a slab without consulting centralized resources 162 (list_lock) where contention may occur. 163 164``slub_min_order`` 165 specifies a minimum order of slabs. A similar effect like 166 ``slub_min_objects``. 167 168``slub_max_order`` 169 specified the order at which ``slub_min_objects`` should no 170 longer be checked. This is useful to avoid SLUB trying to 171 generate super large order pages to fit ``slub_min_objects`` 172 of a slab cache with large object sizes into one high order 173 page. Setting command line parameter 174 ``debug_guardpage_minorder=N`` (N > 0), forces setting 175 ``slub_max_order`` to 0, what cause minimum possible order of 176 slabs allocation. 177 178SLUB Debug output 179================= 180 181Here is a sample of slub debug output:: 182 183 ==================================================================== 184 BUG kmalloc-8: Right Redzone overwritten 185 -------------------------------------------------------------------- 186 187 INFO: 0xc90f6d28-0xc90f6d2b. First byte 0x00 instead of 0xcc 188 INFO: Slab 0xc528c530 flags=0x400000c3 inuse=61 fp=0xc90f6d58 189 INFO: Object 0xc90f6d20 @offset=3360 fp=0xc90f6d58 190 INFO: Allocated in get_modalias+0x61/0xf5 age=53 cpu=1 pid=554 191 192 Bytes b4 (0xc90f6d10): 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ 193 Object (0xc90f6d20): 31 30 31 39 2e 30 30 35 1019.005 194 Redzone (0xc90f6d28): 00 cc cc cc . 195 Padding (0xc90f6d50): 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ 196 197 [<c010523d>] dump_trace+0x63/0x1eb 198 [<c01053df>] show_trace_log_lvl+0x1a/0x2f 199 [<c010601d>] show_trace+0x12/0x14 200 [<c0106035>] dump_stack+0x16/0x18 201 [<c017e0fa>] object_err+0x143/0x14b 202 [<c017e2cc>] check_object+0x66/0x234 203 [<c017eb43>] __slab_free+0x239/0x384 204 [<c017f446>] kfree+0xa6/0xc6 205 [<c02e2335>] get_modalias+0xb9/0xf5 206 [<c02e23b7>] dmi_dev_uevent+0x27/0x3c 207 [<c027866a>] dev_uevent+0x1ad/0x1da 208 [<c0205024>] kobject_uevent_env+0x20a/0x45b 209 [<c020527f>] kobject_uevent+0xa/0xf 210 [<c02779f1>] store_uevent+0x4f/0x58 211 [<c027758e>] dev_attr_store+0x29/0x2f 212 [<c01bec4f>] sysfs_write_file+0x16e/0x19c 213 [<c0183ba7>] vfs_write+0xd1/0x15a 214 [<c01841d7>] sys_write+0x3d/0x72 215 [<c0104112>] sysenter_past_esp+0x5f/0x99 216 [<b7f7b410>] 0xb7f7b410 217 ======================= 218 219 FIX kmalloc-8: Restoring Redzone 0xc90f6d28-0xc90f6d2b=0xcc 220 221If SLUB encounters a corrupted object (full detection requires the kernel 222to be booted with slub_debug) then the following output will be dumped 223into the syslog: 224 2251. Description of the problem encountered 226 227 This will be a message in the system log starting with:: 228 229 =============================================== 230 BUG <slab cache affected>: <What went wrong> 231 ----------------------------------------------- 232 233 INFO: <corruption start>-<corruption_end> <more info> 234 INFO: Slab <address> <slab information> 235 INFO: Object <address> <object information> 236 INFO: Allocated in <kernel function> age=<jiffies since alloc> cpu=<allocated by 237 cpu> pid=<pid of the process> 238 INFO: Freed in <kernel function> age=<jiffies since free> cpu=<freed by cpu> 239 pid=<pid of the process> 240 241 (Object allocation / free information is only available if SLAB_STORE_USER is 242 set for the slab. slub_debug sets that option) 243 2442. The object contents if an object was involved. 245 246 Various types of lines can follow the BUG SLUB line: 247 248 Bytes b4 <address> : <bytes> 249 Shows a few bytes before the object where the problem was detected. 250 Can be useful if the corruption does not stop with the start of the 251 object. 252 253 Object <address> : <bytes> 254 The bytes of the object. If the object is inactive then the bytes 255 typically contain poison values. Any non-poison value shows a 256 corruption by a write after free. 257 258 Redzone <address> : <bytes> 259 The Redzone following the object. The Redzone is used to detect 260 writes after the object. All bytes should always have the same 261 value. If there is any deviation then it is due to a write after 262 the object boundary. 263 264 (Redzone information is only available if SLAB_RED_ZONE is set. 265 slub_debug sets that option) 266 267 Padding <address> : <bytes> 268 Unused data to fill up the space in order to get the next object 269 properly aligned. In the debug case we make sure that there are 270 at least 4 bytes of padding. This allows the detection of writes 271 before the object. 272 2733. A stackdump 274 275 The stackdump describes the location where the error was detected. The cause 276 of the corruption is may be more likely found by looking at the function that 277 allocated or freed the object. 278 2794. Report on how the problem was dealt with in order to ensure the continued 280 operation of the system. 281 282 These are messages in the system log beginning with:: 283 284 FIX <slab cache affected>: <corrective action taken> 285 286 In the above sample SLUB found that the Redzone of an active object has 287 been overwritten. Here a string of 8 characters was written into a slab that 288 has the length of 8 characters. However, a 8 character string needs a 289 terminating 0. That zero has overwritten the first byte of the Redzone field. 290 After reporting the details of the issue encountered the FIX SLUB message 291 tells us that SLUB has restored the Redzone to its proper value and then 292 system operations continue. 293 294Emergency operations 295==================== 296 297Minimal debugging (sanity checks alone) can be enabled by booting with:: 298 299 slub_debug=F 300 301This will be generally be enough to enable the resiliency features of slub 302which will keep the system running even if a bad kernel component will 303keep corrupting objects. This may be important for production systems. 304Performance will be impacted by the sanity checks and there will be a 305continual stream of error messages to the syslog but no additional memory 306will be used (unlike full debugging). 307 308No guarantees. The kernel component still needs to be fixed. Performance 309may be optimized further by locating the slab that experiences corruption 310and enabling debugging only for that cache 311 312I.e.:: 313 314 slub_debug=F,dentry 315 316If the corruption occurs by writing after the end of the object then it 317may be advisable to enable a Redzone to avoid corrupting the beginning 318of other objects:: 319 320 slub_debug=FZ,dentry 321 322Extended slabinfo mode and plotting 323=================================== 324 325The ``slabinfo`` tool has a special 'extended' ('-X') mode that includes: 326 - Slabcache Totals 327 - Slabs sorted by size (up to -N <num> slabs, default 1) 328 - Slabs sorted by loss (up to -N <num> slabs, default 1) 329 330Additionally, in this mode ``slabinfo`` does not dynamically scale 331sizes (G/M/K) and reports everything in bytes (this functionality is 332also available to other slabinfo modes via '-B' option) which makes 333reporting more precise and accurate. Moreover, in some sense the `-X' 334mode also simplifies the analysis of slabs' behaviour, because its 335output can be plotted using the ``slabinfo-gnuplot.sh`` script. So it 336pushes the analysis from looking through the numbers (tons of numbers) 337to something easier -- visual analysis. 338 339To generate plots: 340 341a) collect slabinfo extended records, for example:: 342 343 while [ 1 ]; do slabinfo -X >> FOO_STATS; sleep 1; done 344 345b) pass stats file(-s) to ``slabinfo-gnuplot.sh`` script:: 346 347 slabinfo-gnuplot.sh FOO_STATS [FOO_STATS2 .. FOO_STATSN] 348 349 The ``slabinfo-gnuplot.sh`` script will pre-processes the collected records 350 and generates 3 png files (and 3 pre-processing cache files) per STATS 351 file: 352 - Slabcache Totals: FOO_STATS-totals.png 353 - Slabs sorted by size: FOO_STATS-slabs-by-size.png 354 - Slabs sorted by loss: FOO_STATS-slabs-by-loss.png 355 356Another use case, when ``slabinfo-gnuplot.sh`` can be useful, is when you 357need to compare slabs' behaviour "prior to" and "after" some code 358modification. To help you out there, ``slabinfo-gnuplot.sh`` script 359can 'merge' the `Slabcache Totals` sections from different 360measurements. To visually compare N plots: 361 362a) Collect as many STATS1, STATS2, .. STATSN files as you need:: 363 364 while [ 1 ]; do slabinfo -X >> STATS<X>; sleep 1; done 365 366b) Pre-process those STATS files:: 367 368 slabinfo-gnuplot.sh STATS1 STATS2 .. STATSN 369 370c) Execute ``slabinfo-gnuplot.sh`` in '-t' mode, passing all of the 371 generated pre-processed \*-totals:: 372 373 slabinfo-gnuplot.sh -t STATS1-totals STATS2-totals .. STATSN-totals 374 375 This will produce a single plot (png file). 376 377 Plots, expectedly, can be large so some fluctuations or small spikes 378 can go unnoticed. To deal with that, ``slabinfo-gnuplot.sh`` has two 379 options to 'zoom-in'/'zoom-out': 380 381 a) ``-s %d,%d`` -- overwrites the default image width and height 382 b) ``-r %d,%d`` -- specifies a range of samples to use (for example, 383 in ``slabinfo -X >> FOO_STATS; sleep 1;`` case, using a ``-r 384 40,60`` range will plot only samples collected between 40th and 385 60th seconds). 386 387 388DebugFS files for SLUB 389====================== 390 391For more information about current state of SLUB caches with the user tracking 392debug option enabled, debugfs files are available, typically under 393/sys/kernel/debug/slab/<cache>/ (created only for caches with enabled user 394tracking). There are 2 types of these files with the following debug 395information: 396 3971. alloc_traces:: 398 399 Prints information about unique allocation traces of the currently 400 allocated objects. The output is sorted by frequency of each trace. 401 402 Information in the output: 403 Number of objects, allocating function, minimal/average/maximal jiffies since alloc, 404 pid range of the allocating processes, cpu mask of allocating cpus, and stack trace. 405 406 Example::: 407 408 1085 populate_error_injection_list+0x97/0x110 age=166678/166680/166682 pid=1 cpus=1:: 409 __slab_alloc+0x6d/0x90 410 kmem_cache_alloc_trace+0x2eb/0x300 411 populate_error_injection_list+0x97/0x110 412 init_error_injection+0x1b/0x71 413 do_one_initcall+0x5f/0x2d0 414 kernel_init_freeable+0x26f/0x2d7 415 kernel_init+0xe/0x118 416 ret_from_fork+0x22/0x30 417 418 4192. free_traces:: 420 421 Prints information about unique freeing traces of the currently allocated 422 objects. The freeing traces thus come from the previous life-cycle of the 423 objects and are reported as not available for objects allocated for the first 424 time. The output is sorted by frequency of each trace. 425 426 Information in the output: 427 Number of objects, freeing function, minimal/average/maximal jiffies since free, 428 pid range of the freeing processes, cpu mask of freeing cpus, and stack trace. 429 430 Example::: 431 432 1980 <not-available> age=4294912290 pid=0 cpus=0 433 51 acpi_ut_update_ref_count+0x6a6/0x782 age=236886/237027/237772 pid=1 cpus=1 434 kfree+0x2db/0x420 435 acpi_ut_update_ref_count+0x6a6/0x782 436 acpi_ut_update_object_reference+0x1ad/0x234 437 acpi_ut_remove_reference+0x7d/0x84 438 acpi_rs_get_prt_method_data+0x97/0xd6 439 acpi_get_irq_routing_table+0x82/0xc4 440 acpi_pci_irq_find_prt_entry+0x8e/0x2e0 441 acpi_pci_irq_lookup+0x3a/0x1e0 442 acpi_pci_irq_enable+0x77/0x240 443 pcibios_enable_device+0x39/0x40 444 do_pci_enable_device.part.0+0x5d/0xe0 445 pci_enable_device_flags+0xfc/0x120 446 pci_enable_device+0x13/0x20 447 virtio_pci_probe+0x9e/0x170 448 local_pci_probe+0x48/0x80 449 pci_device_probe+0x105/0x1c0 450 451Christoph Lameter, May 30, 2007 452Sergey Senozhatsky, October 23, 2015 453
