1Kernel Memory Leak Detector 2=========================== 3 4Kmemleak provides a way of detecting possible kernel memory leaks in a 5way similar to a `tracing garbage collector 6<https://en.wikipedia.org/wiki/Tracing_garbage_collection>`_, 7with the difference that the orphan objects are not freed but only 8reported via /sys/kernel/debug/kmemleak. A similar method is used by the 9Valgrind tool (``memcheck --leak-check``) to detect the memory leaks in 10user-space applications. 11 12Usage 13----- 14 15CONFIG_DEBUG_KMEMLEAK in "Kernel hacking" has to be enabled. A kernel 16thread scans the memory every 10 minutes (by default) and prints the 17number of new unreferenced objects found. If the ``debugfs`` isn't already 18mounted, mount with:: 19 20 # mount -t debugfs nodev /sys/kernel/debug/ 21 22To display the details of all the possible scanned memory leaks:: 23 24 # cat /sys/kernel/debug/kmemleak 25 26To trigger an intermediate memory scan:: 27 28 # echo scan > /sys/kernel/debug/kmemleak 29 30To clear the list of all current possible memory leaks:: 31 32 # echo clear > /sys/kernel/debug/kmemleak 33 34New leaks will then come up upon reading ``/sys/kernel/debug/kmemleak`` 35again. 36 37Note that the orphan objects are listed in the order they were allocated 38and one object at the beginning of the list may cause other subsequent 39objects to be reported as orphan. 40 41Memory scanning parameters can be modified at run-time by writing to the 42``/sys/kernel/debug/kmemleak`` file. The following parameters are supported: 43 44- off 45 disable kmemleak (irreversible) 46- stack=on 47 enable the task stacks scanning (default) 48- stack=off 49 disable the tasks stacks scanning 50- scan=on 51 start the automatic memory scanning thread (default) 52- scan=off 53 stop the automatic memory scanning thread 54- scan=<secs> 55 set the automatic memory scanning period in seconds 56 (default 600, 0 to stop the automatic scanning) 57- scan 58 trigger a memory scan 59- clear 60 clear list of current memory leak suspects, done by 61 marking all current reported unreferenced objects grey, 62 or free all kmemleak objects if kmemleak has been disabled. 63- dump=<addr> 64 dump information about the object found at <addr> 65 66Kmemleak can also be disabled at boot-time by passing ``kmemleak=off`` on 67the kernel command line. 68 69Memory may be allocated or freed before kmemleak is initialised, so a 70static pool of metadata objects is used to track those allocations. Once 71kmemleak is fully initialised the pool becomes an emergency reserve, used 72whenever a metadata object cannot be allocated from the slab. The number 73of objects in the pool is configured via the 74CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE option. Exhausting it at run time 75prints "Cannot allocate a kmemleak_object structure" and disables 76kmemleak. 77 78If CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF are enabled, the kmemleak is 79disabled by default. Passing ``kmemleak=on`` on the kernel command 80line enables the function. 81 82If you are getting errors like "Error while writing to stdout" or "write_loop: 83Invalid argument", make sure kmemleak is properly enabled. 84 85Basic Algorithm 86--------------- 87 88The memory allocations via :c:func:`kmalloc`, :c:func:`vmalloc`, 89:c:func:`kmem_cache_alloc` and 90friends are traced and the pointers, together with additional 91information like size and stack trace, are stored in a rbtree. 92The corresponding freeing function calls are tracked and the pointers 93removed from the kmemleak data structures. 94 95An allocated block of memory is considered orphan if no pointer to its 96start address or to any location inside the block can be found by 97scanning the memory (including saved registers). This means that there 98might be no way for the kernel to pass the address of the allocated 99block to a freeing function and therefore the block is considered a 100memory leak. 101 102The scanning algorithm steps: 103 104 1. mark all objects as white (remaining white objects will later be 105 considered orphan) 106 2. scan the memory starting with the data section and stacks, checking 107 the values against the addresses stored in the rbtree. If 108 a pointer to a white object is found, the object is added to the 109 gray list 110 3. scan the gray objects for matching addresses (some white objects 111 can become gray and added at the end of the gray list) until the 112 gray set is finished 113 4. the remaining white objects are considered orphan and reported via 114 /sys/kernel/debug/kmemleak 115 116Some allocated memory blocks have pointers stored in the kernel's 117internal data structures and they cannot be detected as orphans. To 118avoid this, kmemleak can also store the number of values pointing to an 119address inside the block address range that need to be found so that the 120block is not considered a leak. One example is __vmalloc(). 121 122Testing specific sections with kmemleak 123--------------------------------------- 124 125Upon initial bootup your /sys/kernel/debug/kmemleak output page may be 126quite extensive. This can also be the case if you have very buggy code 127when doing development. To work around these situations you can use the 128'clear' command to clear all reported unreferenced objects from the 129/sys/kernel/debug/kmemleak output. By issuing a 'scan' after a 'clear' 130you can find new unreferenced objects; this should help with testing 131specific sections of code. 132 133To test a critical section on demand with a clean kmemleak do:: 134 135 # echo clear > /sys/kernel/debug/kmemleak 136 ... test your kernel or modules ... 137 # echo scan > /sys/kernel/debug/kmemleak 138 139Then as usual to get your report with:: 140 141 # cat /sys/kernel/debug/kmemleak 142 143Freeing kmemleak internal objects 144--------------------------------- 145 146To allow access to previously found memory leaks after kmemleak has been 147disabled by the user or due to an fatal error, internal kmemleak objects 148won't be freed when kmemleak is disabled, and those objects may occupy 149a large part of physical memory. 150 151In this situation, you may reclaim memory with:: 152 153 # echo clear > /sys/kernel/debug/kmemleak 154 155Kmemleak API 156------------ 157 158See the include/linux/kmemleak.h header for the functions prototype. 159 160- ``kmemleak_init`` - initialize kmemleak 161- ``kmemleak_alloc`` - notify of a memory block allocation 162- ``kmemleak_alloc_percpu`` - notify of a percpu memory block allocation 163- ``kmemleak_vmalloc`` - notify of a vmalloc() memory allocation 164- ``kmemleak_free`` - notify of a memory block freeing 165- ``kmemleak_free_part`` - notify of a partial memory block freeing 166- ``kmemleak_free_percpu`` - notify of a percpu memory block freeing 167- ``kmemleak_update_trace`` - update object allocation stack trace 168- ``kmemleak_not_leak`` - mark an object as not a leak 169- ``kmemleak_transient_leak`` - mark an object as a transient leak 170- ``kmemleak_ignore`` - do not scan or report an object as leak 171- ``kmemleak_scan_area`` - add scan areas inside a memory block 172- ``kmemleak_no_scan`` - do not scan a memory block 173- ``kmemleak_erase`` - erase an old value in a pointer variable 174- ``kmemleak_alloc_recursive`` - as kmemleak_alloc but checks the recursiveness 175- ``kmemleak_free_recursive`` - as kmemleak_free but checks the recursiveness 176 177The following functions take a physical address as the object pointer 178and only perform the corresponding action if the address has a lowmem 179mapping: 180 181- ``kmemleak_alloc_phys`` 182- ``kmemleak_free_part_phys`` 183- ``kmemleak_ignore_phys`` 184 185Dealing with false positives/negatives 186-------------------------------------- 187 188The false negatives are real memory leaks (orphan objects) but not 189reported by kmemleak because values found during the memory scanning 190point to such objects. To reduce the number of false negatives, kmemleak 191provides the kmemleak_ignore, kmemleak_scan_area, kmemleak_no_scan and 192kmemleak_erase functions (see above). The task stacks also increase the 193amount of false negatives and their scanning is enabled by default; it 194can be turned off with ``stack=off``. 195 196The false positives are objects wrongly reported as being memory leaks 197(orphan). For objects known not to be leaks, kmemleak provides the 198kmemleak_not_leak function. The kmemleak_ignore could also be used if 199the memory block is known not to contain other pointers and it will no 200longer be scanned. 201 202Some of the reported leaks are only transient, especially on SMP 203systems, because of pointers temporarily stored in CPU registers or 204stacks. Kmemleak defines MSECS_MIN_AGE (defaulting to 5000) representing 205the minimum age of an object to be reported as a memory leak. 206 207The ``min_unref_scans`` module parameter requires an object to be seen 208unreferenced in that many consecutive scans before it is reported. It 209defaults to 3 when CONFIG_DEBUG_KMEMLEAK_VERBOSE is enabled, where the 210periodic scan thread confirms a leak on its own, and to 1 otherwise. A 211value of 1 preserves the historical behaviour; higher values filter the 212transient false positives described above, at the cost of delaying genuine 213reports by up to that many scans. It can be set at boot with 214``kmemleak.min_unref_scans=<n>`` or at run-time via 215``/sys/module/kmemleak/parameters/min_unref_scans``. 216 217Limitations and Drawbacks 218------------------------- 219 220The main drawback is the reduced performance of memory allocation and 221freeing. To avoid other penalties, the memory scanning is performed by a 222periodic thread rather than on every allocation. Reading the 223/sys/kernel/debug/kmemleak file only lists the objects found by the last 224scan; writing ``scan`` to it triggers a new one. Anyway, this tool is 225intended for debugging purposes where the performance might not be the 226most important requirement. 227 228To keep the algorithm simple, kmemleak scans for values pointing to any 229address inside a block's address range. This may lead to an increased 230number of false negatives. However, it is likely that a real memory leak 231will eventually become visible. 232 233Another source of false negatives is the data stored in non-pointer 234values. In a future version, kmemleak could only scan the pointer 235members in the allocated structures. This feature would solve many of 236the false negative cases described above. 237 238The tool can report false positives. These are cases where an allocated 239block doesn't need to be freed (some cases in the init_call functions), 240the pointer is calculated by other methods than the usual container_of 241macro or the pointer is stored in a location not scanned by kmemleak. 242 243Page allocations and ioremap are not tracked. 244 245Testing with kmemleak-test 246-------------------------- 247 248To check if you have all set up to use kmemleak, you can use the kmemleak-test 249module, a module that deliberately leaks memory. Set CONFIG_SAMPLE_KMEMLEAK 250as module (it can't be used as built-in) and boot the kernel with kmemleak 251enabled. Load the module and perform a scan with:: 252 253 # modprobe kmemleak-test 254 # echo scan > /sys/kernel/debug/kmemleak 255 256Note that the you may not get results instantly or on the first scanning. When 257kmemleak gets results, it'll log ``kmemleak: <count of leaks> new suspected 258memory leaks``. Then read the file to see then:: 259 260 # cat /sys/kernel/debug/kmemleak 261 unreferenced object 0xffff89862ca702e8 (size 32): 262 comm "modprobe", pid 2088, jiffies 4294680594 (age 375.486s) 263 hex dump (first 32 bytes): 264 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk 265 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk. 266 backtrace: 267 [<00000000e0a73ec7>] 0xffffffffc01d2036 268 [<000000000c5d2a46>] do_one_initcall+0x41/0x1df 269 [<0000000046db7e0a>] do_init_module+0x55/0x200 270 [<00000000542b9814>] load_module+0x203c/0x2480 271 [<00000000c2850256>] __do_sys_finit_module+0xba/0xe0 272 [<000000006564e7ef>] do_syscall_64+0x43/0x110 273 [<000000007c873fa6>] entry_SYSCALL_64_after_hwframe+0x44/0xa9 274 ... 275 276Removing the module with ``rmmod kmemleak_test`` should also trigger some 277kmemleak results. 278