110ffebbeSMauro Carvalho Chehab=========================================== 210ffebbeSMauro Carvalho ChehabFault injection capabilities infrastructure 310ffebbeSMauro Carvalho Chehab=========================================== 410ffebbeSMauro Carvalho Chehab 510ffebbeSMauro Carvalho ChehabSee also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug. 610ffebbeSMauro Carvalho Chehab 710ffebbeSMauro Carvalho Chehab 810ffebbeSMauro Carvalho ChehabAvailable fault injection capabilities 910ffebbeSMauro Carvalho Chehab-------------------------------------- 1010ffebbeSMauro Carvalho Chehab 1110ffebbeSMauro Carvalho Chehab- failslab 1210ffebbeSMauro Carvalho Chehab 1310ffebbeSMauro Carvalho Chehab injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...) 1410ffebbeSMauro Carvalho Chehab 1510ffebbeSMauro Carvalho Chehab- fail_page_alloc 1610ffebbeSMauro Carvalho Chehab 1710ffebbeSMauro Carvalho Chehab injects page allocation failures. (alloc_pages(), get_free_pages(), ...) 1810ffebbeSMauro Carvalho Chehab 192c739cedSAlbert van der Linde- fail_usercopy 202c739cedSAlbert van der Linde 212c739cedSAlbert van der Linde injects failures in user memory access functions. (copy_from_user(), get_user(), ...) 222c739cedSAlbert van der Linde 2310ffebbeSMauro Carvalho Chehab- fail_futex 2410ffebbeSMauro Carvalho Chehab 2510ffebbeSMauro Carvalho Chehab injects futex deadlock and uaddr fault errors. 2610ffebbeSMauro Carvalho Chehab 27400edd8cSChuck Lever- fail_sunrpc 28400edd8cSChuck Lever 29400edd8cSChuck Lever injects kernel RPC client and server failures. 30400edd8cSChuck Lever 3110ffebbeSMauro Carvalho Chehab- fail_make_request 3210ffebbeSMauro Carvalho Chehab 3310ffebbeSMauro Carvalho Chehab injects disk IO errors on devices permitted by setting 3410ffebbeSMauro Carvalho Chehab /sys/block/<device>/make-it-fail or 35ed00aabdSChristoph Hellwig /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct()) 3610ffebbeSMauro Carvalho Chehab 3710ffebbeSMauro Carvalho Chehab- fail_mmc_request 3810ffebbeSMauro Carvalho Chehab 3910ffebbeSMauro Carvalho Chehab injects MMC data errors on devices permitted by setting 4010ffebbeSMauro Carvalho Chehab debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request 4110ffebbeSMauro Carvalho Chehab 4210ffebbeSMauro Carvalho Chehab- fail_function 4310ffebbeSMauro Carvalho Chehab 4410ffebbeSMauro Carvalho Chehab injects error return on specific functions, which are marked by 4510ffebbeSMauro Carvalho Chehab ALLOW_ERROR_INJECTION() macro, by setting debugfs entries 4610ffebbeSMauro Carvalho Chehab under /sys/kernel/debug/fail_function. No boot option supported. 4710ffebbeSMauro Carvalho Chehab 4810ffebbeSMauro Carvalho Chehab- NVMe fault injection 4910ffebbeSMauro Carvalho Chehab 5010ffebbeSMauro Carvalho Chehab inject NVMe status code and retry flag on devices permitted by setting 5110ffebbeSMauro Carvalho Chehab debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default 5210ffebbeSMauro Carvalho Chehab status code is NVME_SC_INVALID_OPCODE with no retry. The status code and 5310ffebbeSMauro Carvalho Chehab retry flag can be set via the debugfs. 5410ffebbeSMauro Carvalho Chehab 5510ffebbeSMauro Carvalho Chehab 5610ffebbeSMauro Carvalho ChehabConfigure fault-injection capabilities behavior 5710ffebbeSMauro Carvalho Chehab----------------------------------------------- 5810ffebbeSMauro Carvalho Chehab 5910ffebbeSMauro Carvalho Chehabdebugfs entries 6010ffebbeSMauro Carvalho Chehab^^^^^^^^^^^^^^^ 6110ffebbeSMauro Carvalho Chehab 6210ffebbeSMauro Carvalho Chehabfault-inject-debugfs kernel module provides some debugfs entries for runtime 6310ffebbeSMauro Carvalho Chehabconfiguration of fault-injection capabilities. 6410ffebbeSMauro Carvalho Chehab 6510ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail*/probability: 6610ffebbeSMauro Carvalho Chehab 6710ffebbeSMauro Carvalho Chehab likelihood of failure injection, in percent. 6810ffebbeSMauro Carvalho Chehab 6910ffebbeSMauro Carvalho Chehab Format: <percent> 7010ffebbeSMauro Carvalho Chehab 7110ffebbeSMauro Carvalho Chehab Note that one-failure-per-hundred is a very high error rate 7210ffebbeSMauro Carvalho Chehab for some testcases. Consider setting probability=100 and configure 7310ffebbeSMauro Carvalho Chehab /sys/kernel/debug/fail*/interval for such testcases. 7410ffebbeSMauro Carvalho Chehab 7510ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail*/interval: 7610ffebbeSMauro Carvalho Chehab 7710ffebbeSMauro Carvalho Chehab specifies the interval between failures, for calls to 7810ffebbeSMauro Carvalho Chehab should_fail() that pass all the other tests. 7910ffebbeSMauro Carvalho Chehab 8010ffebbeSMauro Carvalho Chehab Note that if you enable this, by setting interval>1, you will 8110ffebbeSMauro Carvalho Chehab probably want to set probability=100. 8210ffebbeSMauro Carvalho Chehab 8310ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail*/times: 8410ffebbeSMauro Carvalho Chehab 8500574752SWolfram Sang specifies how many times failures may happen at most. A value of -1 8600574752SWolfram Sang means "no limit". Note, though, that this file only accepts unsigned 8700574752SWolfram Sang values. So, if you want to specify -1, you better use 'printf' instead 8800574752SWolfram Sang of 'echo', e.g.: $ printf %#x -1 > times 8910ffebbeSMauro Carvalho Chehab 9010ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail*/space: 9110ffebbeSMauro Carvalho Chehab 9210ffebbeSMauro Carvalho Chehab specifies an initial resource "budget", decremented by "size" 9310ffebbeSMauro Carvalho Chehab on each call to should_fail(,size). Failure injection is 9410ffebbeSMauro Carvalho Chehab suppressed until "space" reaches zero. 9510ffebbeSMauro Carvalho Chehab 9610ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail*/verbose 9710ffebbeSMauro Carvalho Chehab 9810ffebbeSMauro Carvalho Chehab Format: { 0 | 1 | 2 } 9910ffebbeSMauro Carvalho Chehab 10010ffebbeSMauro Carvalho Chehab specifies the verbosity of the messages when failure is 10110ffebbeSMauro Carvalho Chehab injected. '0' means no messages; '1' will print only a single 10210ffebbeSMauro Carvalho Chehab log line per failure; '2' will print a call trace too -- useful 10310ffebbeSMauro Carvalho Chehab to debug the problems revealed by fault injection. 10410ffebbeSMauro Carvalho Chehab 10510ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail*/task-filter: 10610ffebbeSMauro Carvalho Chehab 10710ffebbeSMauro Carvalho Chehab Format: { 'Y' | 'N' } 10810ffebbeSMauro Carvalho Chehab 10910ffebbeSMauro Carvalho Chehab A value of 'N' disables filtering by process (default). 11010ffebbeSMauro Carvalho Chehab Any positive value limits failures to only processes indicated by 11110ffebbeSMauro Carvalho Chehab /proc/<pid>/make-it-fail==1. 11210ffebbeSMauro Carvalho Chehab 11310ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail*/require-start, 11410ffebbeSMauro Carvalho Chehab /sys/kernel/debug/fail*/require-end, 11510ffebbeSMauro Carvalho Chehab /sys/kernel/debug/fail*/reject-start, 11610ffebbeSMauro Carvalho Chehab /sys/kernel/debug/fail*/reject-end: 11710ffebbeSMauro Carvalho Chehab 11810ffebbeSMauro Carvalho Chehab specifies the range of virtual addresses tested during 11910ffebbeSMauro Carvalho Chehab stacktrace walking. Failure is injected only if some caller 12010ffebbeSMauro Carvalho Chehab in the walked stacktrace lies within the required range, and 12110ffebbeSMauro Carvalho Chehab none lies within the rejected range. 12210ffebbeSMauro Carvalho Chehab Default required range is [0,ULONG_MAX) (whole of virtual address space). 12310ffebbeSMauro Carvalho Chehab Default rejected range is [0,0). 12410ffebbeSMauro Carvalho Chehab 12510ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail*/stacktrace-depth: 12610ffebbeSMauro Carvalho Chehab 12710ffebbeSMauro Carvalho Chehab specifies the maximum stacktrace depth walked during search 12810ffebbeSMauro Carvalho Chehab for a caller within [require-start,require-end) OR 12910ffebbeSMauro Carvalho Chehab [reject-start,reject-end). 13010ffebbeSMauro Carvalho Chehab 13110ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem: 13210ffebbeSMauro Carvalho Chehab 13310ffebbeSMauro Carvalho Chehab Format: { 'Y' | 'N' } 13410ffebbeSMauro Carvalho Chehab 135bad3fbb2SDylan Yudaken default is 'Y', setting it to 'N' will also inject failures into 136bad3fbb2SDylan Yudaken highmem/user allocations (__GFP_HIGHMEM allocations). 13710ffebbeSMauro Carvalho Chehab 13810ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/failslab/ignore-gfp-wait: 13910ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait: 14010ffebbeSMauro Carvalho Chehab 14110ffebbeSMauro Carvalho Chehab Format: { 'Y' | 'N' } 14210ffebbeSMauro Carvalho Chehab 143bad3fbb2SDylan Yudaken default is 'Y', setting it to 'N' will also inject failures 144bad3fbb2SDylan Yudaken into allocations that can sleep (__GFP_DIRECT_RECLAIM allocations). 14510ffebbeSMauro Carvalho Chehab 14610ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail_page_alloc/min-order: 14710ffebbeSMauro Carvalho Chehab 14810ffebbeSMauro Carvalho Chehab specifies the minimum page allocation order to be injected 14910ffebbeSMauro Carvalho Chehab failures. 15010ffebbeSMauro Carvalho Chehab 15110ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail_futex/ignore-private: 15210ffebbeSMauro Carvalho Chehab 15310ffebbeSMauro Carvalho Chehab Format: { 'Y' | 'N' } 15410ffebbeSMauro Carvalho Chehab 15510ffebbeSMauro Carvalho Chehab default is 'N', setting it to 'Y' will disable failure injections 15610ffebbeSMauro Carvalho Chehab when dealing with private (address space) futexes. 15710ffebbeSMauro Carvalho Chehab 158400edd8cSChuck Lever- /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect: 159400edd8cSChuck Lever 160400edd8cSChuck Lever Format: { 'Y' | 'N' } 161400edd8cSChuck Lever 162400edd8cSChuck Lever default is 'N', setting it to 'Y' will disable disconnect 163400edd8cSChuck Lever injection on the RPC client. 164400edd8cSChuck Lever 165400edd8cSChuck Lever- /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect: 166400edd8cSChuck Lever 167400edd8cSChuck Lever Format: { 'Y' | 'N' } 168400edd8cSChuck Lever 169400edd8cSChuck Lever default is 'N', setting it to 'Y' will disable disconnect 170400edd8cSChuck Lever injection on the RPC server. 171400edd8cSChuck Lever 172*36f2ef2dSChuck Lever- /sys/kernel/debug/fail_sunrpc/ignore-cache-wait: 173*36f2ef2dSChuck Lever 174*36f2ef2dSChuck Lever Format: { 'Y' | 'N' } 175*36f2ef2dSChuck Lever 176*36f2ef2dSChuck Lever default is 'N', setting it to 'Y' will disable cache wait 177*36f2ef2dSChuck Lever injection on the RPC server. 178*36f2ef2dSChuck Lever 17910ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail_function/inject: 18010ffebbeSMauro Carvalho Chehab 18110ffebbeSMauro Carvalho Chehab Format: { 'function-name' | '!function-name' | '' } 18210ffebbeSMauro Carvalho Chehab 18310ffebbeSMauro Carvalho Chehab specifies the target function of error injection by name. 18410ffebbeSMauro Carvalho Chehab If the function name leads '!' prefix, given function is 18510ffebbeSMauro Carvalho Chehab removed from injection list. If nothing specified ('') 18610ffebbeSMauro Carvalho Chehab injection list is cleared. 18710ffebbeSMauro Carvalho Chehab 18810ffebbeSMauro Carvalho Chehab- /sys/kernel/debug/fail_function/injectable: 18910ffebbeSMauro Carvalho Chehab 19010ffebbeSMauro Carvalho Chehab (read only) shows error injectable functions and what type of 19110ffebbeSMauro Carvalho Chehab error values can be specified. The error type will be one of 19210ffebbeSMauro Carvalho Chehab below; 19310ffebbeSMauro Carvalho Chehab - NULL: retval must be 0. 19410ffebbeSMauro Carvalho Chehab - ERRNO: retval must be -1 to -MAX_ERRNO (-4096). 19510ffebbeSMauro Carvalho Chehab - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096). 19610ffebbeSMauro Carvalho Chehab 19700574752SWolfram Sang- /sys/kernel/debug/fail_function/<function-name>/retval: 19810ffebbeSMauro Carvalho Chehab 19900574752SWolfram Sang specifies the "error" return value to inject to the given function. 20000574752SWolfram Sang This will be created when the user specifies a new injection entry. 20100574752SWolfram Sang Note that this file only accepts unsigned values. So, if you want to 20200574752SWolfram Sang use a negative errno, you better use 'printf' instead of 'echo', e.g.: 20300574752SWolfram Sang $ printf %#x -12 > retval 20410ffebbeSMauro Carvalho Chehab 20510ffebbeSMauro Carvalho ChehabBoot option 20610ffebbeSMauro Carvalho Chehab^^^^^^^^^^^ 20710ffebbeSMauro Carvalho Chehab 20810ffebbeSMauro Carvalho ChehabIn order to inject faults while debugfs is not available (early boot time), 20910ffebbeSMauro Carvalho Chehabuse the boot option:: 21010ffebbeSMauro Carvalho Chehab 21110ffebbeSMauro Carvalho Chehab failslab= 21210ffebbeSMauro Carvalho Chehab fail_page_alloc= 2132c739cedSAlbert van der Linde fail_usercopy= 21410ffebbeSMauro Carvalho Chehab fail_make_request= 21510ffebbeSMauro Carvalho Chehab fail_futex= 21610ffebbeSMauro Carvalho Chehab mmc_core.fail_request=<interval>,<probability>,<space>,<times> 21710ffebbeSMauro Carvalho Chehab 21810ffebbeSMauro Carvalho Chehabproc entries 21910ffebbeSMauro Carvalho Chehab^^^^^^^^^^^^ 22010ffebbeSMauro Carvalho Chehab 22110ffebbeSMauro Carvalho Chehab- /proc/<pid>/fail-nth, 22210ffebbeSMauro Carvalho Chehab /proc/self/task/<tid>/fail-nth: 22310ffebbeSMauro Carvalho Chehab 22410ffebbeSMauro Carvalho Chehab Write to this file of integer N makes N-th call in the task fail. 22510ffebbeSMauro Carvalho Chehab Read from this file returns a integer value. A value of '0' indicates 22610ffebbeSMauro Carvalho Chehab that the fault setup with a previous write to this file was injected. 22710ffebbeSMauro Carvalho Chehab A positive integer N indicates that the fault wasn't yet injected. 22810ffebbeSMauro Carvalho Chehab Note that this file enables all types of faults (slab, futex, etc). 22910ffebbeSMauro Carvalho Chehab This setting takes precedence over all other generic debugfs settings 23010ffebbeSMauro Carvalho Chehab like probability, interval, times, etc. But per-capability settings 23110ffebbeSMauro Carvalho Chehab (e.g. fail_futex/ignore-private) take precedence over it. 23210ffebbeSMauro Carvalho Chehab 23310ffebbeSMauro Carvalho Chehab This feature is intended for systematic testing of faults in a single 23410ffebbeSMauro Carvalho Chehab system call. See an example below. 23510ffebbeSMauro Carvalho Chehab 23610ffebbeSMauro Carvalho ChehabHow to add new fault injection capability 23710ffebbeSMauro Carvalho Chehab----------------------------------------- 23810ffebbeSMauro Carvalho Chehab 23910ffebbeSMauro Carvalho Chehab- #include <linux/fault-inject.h> 24010ffebbeSMauro Carvalho Chehab 24110ffebbeSMauro Carvalho Chehab- define the fault attributes 24210ffebbeSMauro Carvalho Chehab 24310ffebbeSMauro Carvalho Chehab DECLARE_FAULT_ATTR(name); 24410ffebbeSMauro Carvalho Chehab 24510ffebbeSMauro Carvalho Chehab Please see the definition of struct fault_attr in fault-inject.h 24610ffebbeSMauro Carvalho Chehab for details. 24710ffebbeSMauro Carvalho Chehab 24810ffebbeSMauro Carvalho Chehab- provide a way to configure fault attributes 24910ffebbeSMauro Carvalho Chehab 25010ffebbeSMauro Carvalho Chehab- boot option 25110ffebbeSMauro Carvalho Chehab 25210ffebbeSMauro Carvalho Chehab If you need to enable the fault injection capability from boot time, you can 25310ffebbeSMauro Carvalho Chehab provide boot option to configure it. There is a helper function for it: 25410ffebbeSMauro Carvalho Chehab 25510ffebbeSMauro Carvalho Chehab setup_fault_attr(attr, str); 25610ffebbeSMauro Carvalho Chehab 25710ffebbeSMauro Carvalho Chehab- debugfs entries 25810ffebbeSMauro Carvalho Chehab 2592c739cedSAlbert van der Linde failslab, fail_page_alloc, fail_usercopy, and fail_make_request use this way. 26010ffebbeSMauro Carvalho Chehab Helper functions: 26110ffebbeSMauro Carvalho Chehab 26210ffebbeSMauro Carvalho Chehab fault_create_debugfs_attr(name, parent, attr); 26310ffebbeSMauro Carvalho Chehab 26410ffebbeSMauro Carvalho Chehab- module parameters 26510ffebbeSMauro Carvalho Chehab 26610ffebbeSMauro Carvalho Chehab If the scope of the fault injection capability is limited to a 26710ffebbeSMauro Carvalho Chehab single kernel module, it is better to provide module parameters to 26810ffebbeSMauro Carvalho Chehab configure the fault attributes. 26910ffebbeSMauro Carvalho Chehab 27010ffebbeSMauro Carvalho Chehab- add a hook to insert failures 27110ffebbeSMauro Carvalho Chehab 27210ffebbeSMauro Carvalho Chehab Upon should_fail() returning true, client code should inject a failure: 27310ffebbeSMauro Carvalho Chehab 27410ffebbeSMauro Carvalho Chehab should_fail(attr, size); 27510ffebbeSMauro Carvalho Chehab 27610ffebbeSMauro Carvalho ChehabApplication Examples 27710ffebbeSMauro Carvalho Chehab-------------------- 27810ffebbeSMauro Carvalho Chehab 27910ffebbeSMauro Carvalho Chehab- Inject slab allocation failures into module init/exit code:: 28010ffebbeSMauro Carvalho Chehab 28110ffebbeSMauro Carvalho Chehab #!/bin/bash 28210ffebbeSMauro Carvalho Chehab 28310ffebbeSMauro Carvalho Chehab FAILTYPE=failslab 28410ffebbeSMauro Carvalho Chehab echo Y > /sys/kernel/debug/$FAILTYPE/task-filter 28510ffebbeSMauro Carvalho Chehab echo 10 > /sys/kernel/debug/$FAILTYPE/probability 28610ffebbeSMauro Carvalho Chehab echo 100 > /sys/kernel/debug/$FAILTYPE/interval 28700574752SWolfram Sang printf %#x -1 > /sys/kernel/debug/$FAILTYPE/times 28810ffebbeSMauro Carvalho Chehab echo 0 > /sys/kernel/debug/$FAILTYPE/space 28910ffebbeSMauro Carvalho Chehab echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 290bad3fbb2SDylan Yudaken echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 29110ffebbeSMauro Carvalho Chehab 29210ffebbeSMauro Carvalho Chehab faulty_system() 29310ffebbeSMauro Carvalho Chehab { 29410ffebbeSMauro Carvalho Chehab bash -c "echo 1 > /proc/self/make-it-fail && exec $*" 29510ffebbeSMauro Carvalho Chehab } 29610ffebbeSMauro Carvalho Chehab 29710ffebbeSMauro Carvalho Chehab if [ $# -eq 0 ] 29810ffebbeSMauro Carvalho Chehab then 29910ffebbeSMauro Carvalho Chehab echo "Usage: $0 modulename [ modulename ... ]" 30010ffebbeSMauro Carvalho Chehab exit 1 30110ffebbeSMauro Carvalho Chehab fi 30210ffebbeSMauro Carvalho Chehab 30310ffebbeSMauro Carvalho Chehab for m in $* 30410ffebbeSMauro Carvalho Chehab do 30510ffebbeSMauro Carvalho Chehab echo inserting $m... 30610ffebbeSMauro Carvalho Chehab faulty_system modprobe $m 30710ffebbeSMauro Carvalho Chehab 30810ffebbeSMauro Carvalho Chehab echo removing $m... 30910ffebbeSMauro Carvalho Chehab faulty_system modprobe -r $m 31010ffebbeSMauro Carvalho Chehab done 31110ffebbeSMauro Carvalho Chehab 31210ffebbeSMauro Carvalho Chehab------------------------------------------------------------------------------ 31310ffebbeSMauro Carvalho Chehab 31410ffebbeSMauro Carvalho Chehab- Inject page allocation failures only for a specific module:: 31510ffebbeSMauro Carvalho Chehab 31610ffebbeSMauro Carvalho Chehab #!/bin/bash 31710ffebbeSMauro Carvalho Chehab 31810ffebbeSMauro Carvalho Chehab FAILTYPE=fail_page_alloc 31910ffebbeSMauro Carvalho Chehab module=$1 32010ffebbeSMauro Carvalho Chehab 32110ffebbeSMauro Carvalho Chehab if [ -z $module ] 32210ffebbeSMauro Carvalho Chehab then 32310ffebbeSMauro Carvalho Chehab echo "Usage: $0 <modulename>" 32410ffebbeSMauro Carvalho Chehab exit 1 32510ffebbeSMauro Carvalho Chehab fi 32610ffebbeSMauro Carvalho Chehab 32710ffebbeSMauro Carvalho Chehab modprobe $module 32810ffebbeSMauro Carvalho Chehab 32910ffebbeSMauro Carvalho Chehab if [ ! -d /sys/module/$module/sections ] 33010ffebbeSMauro Carvalho Chehab then 33110ffebbeSMauro Carvalho Chehab echo Module $module is not loaded 33210ffebbeSMauro Carvalho Chehab exit 1 33310ffebbeSMauro Carvalho Chehab fi 33410ffebbeSMauro Carvalho Chehab 33510ffebbeSMauro Carvalho Chehab cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start 33610ffebbeSMauro Carvalho Chehab cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end 33710ffebbeSMauro Carvalho Chehab 33810ffebbeSMauro Carvalho Chehab echo N > /sys/kernel/debug/$FAILTYPE/task-filter 33910ffebbeSMauro Carvalho Chehab echo 10 > /sys/kernel/debug/$FAILTYPE/probability 34010ffebbeSMauro Carvalho Chehab echo 100 > /sys/kernel/debug/$FAILTYPE/interval 34100574752SWolfram Sang printf %#x -1 > /sys/kernel/debug/$FAILTYPE/times 34210ffebbeSMauro Carvalho Chehab echo 0 > /sys/kernel/debug/$FAILTYPE/space 34310ffebbeSMauro Carvalho Chehab echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 344bad3fbb2SDylan Yudaken echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 345bad3fbb2SDylan Yudaken echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem 34610ffebbeSMauro Carvalho Chehab echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth 34710ffebbeSMauro Carvalho Chehab 34810ffebbeSMauro Carvalho Chehab trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT 34910ffebbeSMauro Carvalho Chehab 35010ffebbeSMauro Carvalho Chehab echo "Injecting errors into the module $module... (interrupt to stop)" 35110ffebbeSMauro Carvalho Chehab sleep 1000000 35210ffebbeSMauro Carvalho Chehab 35310ffebbeSMauro Carvalho Chehab------------------------------------------------------------------------------ 35410ffebbeSMauro Carvalho Chehab 35510ffebbeSMauro Carvalho Chehab- Inject open_ctree error while btrfs mount:: 35610ffebbeSMauro Carvalho Chehab 35710ffebbeSMauro Carvalho Chehab #!/bin/bash 35810ffebbeSMauro Carvalho Chehab 35910ffebbeSMauro Carvalho Chehab rm -f testfile.img 36010ffebbeSMauro Carvalho Chehab dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1 36110ffebbeSMauro Carvalho Chehab DEVICE=$(losetup --show -f testfile.img) 36210ffebbeSMauro Carvalho Chehab mkfs.btrfs -f $DEVICE 36310ffebbeSMauro Carvalho Chehab mkdir -p tmpmnt 36410ffebbeSMauro Carvalho Chehab 36510ffebbeSMauro Carvalho Chehab FAILTYPE=fail_function 36610ffebbeSMauro Carvalho Chehab FAILFUNC=open_ctree 36710ffebbeSMauro Carvalho Chehab echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject 36800574752SWolfram Sang printf %#x -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval 36910ffebbeSMauro Carvalho Chehab echo N > /sys/kernel/debug/$FAILTYPE/task-filter 37010ffebbeSMauro Carvalho Chehab echo 100 > /sys/kernel/debug/$FAILTYPE/probability 37110ffebbeSMauro Carvalho Chehab echo 0 > /sys/kernel/debug/$FAILTYPE/interval 37200574752SWolfram Sang printf %#x -1 > /sys/kernel/debug/$FAILTYPE/times 37310ffebbeSMauro Carvalho Chehab echo 0 > /sys/kernel/debug/$FAILTYPE/space 37410ffebbeSMauro Carvalho Chehab echo 1 > /sys/kernel/debug/$FAILTYPE/verbose 37510ffebbeSMauro Carvalho Chehab 37610ffebbeSMauro Carvalho Chehab mount -t btrfs $DEVICE tmpmnt 37710ffebbeSMauro Carvalho Chehab if [ $? -ne 0 ] 37810ffebbeSMauro Carvalho Chehab then 37910ffebbeSMauro Carvalho Chehab echo "SUCCESS!" 38010ffebbeSMauro Carvalho Chehab else 38110ffebbeSMauro Carvalho Chehab echo "FAILED!" 38210ffebbeSMauro Carvalho Chehab umount tmpmnt 38310ffebbeSMauro Carvalho Chehab fi 38410ffebbeSMauro Carvalho Chehab 38510ffebbeSMauro Carvalho Chehab echo > /sys/kernel/debug/$FAILTYPE/inject 38610ffebbeSMauro Carvalho Chehab 38710ffebbeSMauro Carvalho Chehab rmdir tmpmnt 38810ffebbeSMauro Carvalho Chehab losetup -d $DEVICE 38910ffebbeSMauro Carvalho Chehab rm testfile.img 39010ffebbeSMauro Carvalho Chehab 39110ffebbeSMauro Carvalho Chehab 39210ffebbeSMauro Carvalho ChehabTool to run command with failslab or fail_page_alloc 39310ffebbeSMauro Carvalho Chehab---------------------------------------------------- 39410ffebbeSMauro Carvalho ChehabIn order to make it easier to accomplish the tasks mentioned above, we can use 39510ffebbeSMauro Carvalho Chehabtools/testing/fault-injection/failcmd.sh. Please run a command 39610ffebbeSMauro Carvalho Chehab"./tools/testing/fault-injection/failcmd.sh --help" for more information and 39710ffebbeSMauro Carvalho Chehabsee the following examples. 39810ffebbeSMauro Carvalho Chehab 39910ffebbeSMauro Carvalho ChehabExamples: 40010ffebbeSMauro Carvalho Chehab 40110ffebbeSMauro Carvalho ChehabRun a command "make -C tools/testing/selftests/ run_tests" with injecting slab 40210ffebbeSMauro Carvalho Chehaballocation failure:: 40310ffebbeSMauro Carvalho Chehab 40410ffebbeSMauro Carvalho Chehab # ./tools/testing/fault-injection/failcmd.sh \ 40510ffebbeSMauro Carvalho Chehab -- make -C tools/testing/selftests/ run_tests 40610ffebbeSMauro Carvalho Chehab 40710ffebbeSMauro Carvalho ChehabSame as above except to specify 100 times failures at most instead of one time 40810ffebbeSMauro Carvalho Chehabat most by default:: 40910ffebbeSMauro Carvalho Chehab 41010ffebbeSMauro Carvalho Chehab # ./tools/testing/fault-injection/failcmd.sh --times=100 \ 41110ffebbeSMauro Carvalho Chehab -- make -C tools/testing/selftests/ run_tests 41210ffebbeSMauro Carvalho Chehab 41310ffebbeSMauro Carvalho ChehabSame as above except to inject page allocation failure instead of slab 41410ffebbeSMauro Carvalho Chehaballocation failure:: 41510ffebbeSMauro Carvalho Chehab 41610ffebbeSMauro Carvalho Chehab # env FAILCMD_TYPE=fail_page_alloc \ 41710ffebbeSMauro Carvalho Chehab ./tools/testing/fault-injection/failcmd.sh --times=100 \ 41810ffebbeSMauro Carvalho Chehab -- make -C tools/testing/selftests/ run_tests 41910ffebbeSMauro Carvalho Chehab 42010ffebbeSMauro Carvalho ChehabSystematic faults using fail-nth 42110ffebbeSMauro Carvalho Chehab--------------------------------- 42210ffebbeSMauro Carvalho Chehab 42310ffebbeSMauro Carvalho ChehabThe following code systematically faults 0-th, 1-st, 2-nd and so on 42410ffebbeSMauro Carvalho Chehabcapabilities in the socketpair() system call:: 42510ffebbeSMauro Carvalho Chehab 42610ffebbeSMauro Carvalho Chehab #include <sys/types.h> 42710ffebbeSMauro Carvalho Chehab #include <sys/stat.h> 42810ffebbeSMauro Carvalho Chehab #include <sys/socket.h> 42910ffebbeSMauro Carvalho Chehab #include <sys/syscall.h> 43010ffebbeSMauro Carvalho Chehab #include <fcntl.h> 43110ffebbeSMauro Carvalho Chehab #include <unistd.h> 43210ffebbeSMauro Carvalho Chehab #include <string.h> 43310ffebbeSMauro Carvalho Chehab #include <stdlib.h> 43410ffebbeSMauro Carvalho Chehab #include <stdio.h> 43510ffebbeSMauro Carvalho Chehab #include <errno.h> 43610ffebbeSMauro Carvalho Chehab 43710ffebbeSMauro Carvalho Chehab int main() 43810ffebbeSMauro Carvalho Chehab { 43910ffebbeSMauro Carvalho Chehab int i, err, res, fail_nth, fds[2]; 44010ffebbeSMauro Carvalho Chehab char buf[128]; 44110ffebbeSMauro Carvalho Chehab 44210ffebbeSMauro Carvalho Chehab system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait"); 44310ffebbeSMauro Carvalho Chehab sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid)); 44410ffebbeSMauro Carvalho Chehab fail_nth = open(buf, O_RDWR); 44510ffebbeSMauro Carvalho Chehab for (i = 1;; i++) { 44610ffebbeSMauro Carvalho Chehab sprintf(buf, "%d", i); 44710ffebbeSMauro Carvalho Chehab write(fail_nth, buf, strlen(buf)); 44810ffebbeSMauro Carvalho Chehab res = socketpair(AF_LOCAL, SOCK_STREAM, 0, fds); 44910ffebbeSMauro Carvalho Chehab err = errno; 45010ffebbeSMauro Carvalho Chehab pread(fail_nth, buf, sizeof(buf), 0); 45110ffebbeSMauro Carvalho Chehab if (res == 0) { 45210ffebbeSMauro Carvalho Chehab close(fds[0]); 45310ffebbeSMauro Carvalho Chehab close(fds[1]); 45410ffebbeSMauro Carvalho Chehab } 45510ffebbeSMauro Carvalho Chehab printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y', 45610ffebbeSMauro Carvalho Chehab res, err); 45710ffebbeSMauro Carvalho Chehab if (atoi(buf)) 45810ffebbeSMauro Carvalho Chehab break; 45910ffebbeSMauro Carvalho Chehab } 46010ffebbeSMauro Carvalho Chehab return 0; 46110ffebbeSMauro Carvalho Chehab } 46210ffebbeSMauro Carvalho Chehab 46310ffebbeSMauro Carvalho ChehabAn example output:: 46410ffebbeSMauro Carvalho Chehab 46510ffebbeSMauro Carvalho Chehab 1-th fault Y: res=-1/23 46610ffebbeSMauro Carvalho Chehab 2-th fault Y: res=-1/23 46710ffebbeSMauro Carvalho Chehab 3-th fault Y: res=-1/12 46810ffebbeSMauro Carvalho Chehab 4-th fault Y: res=-1/12 46910ffebbeSMauro Carvalho Chehab 5-th fault Y: res=-1/23 47010ffebbeSMauro Carvalho Chehab 6-th fault Y: res=-1/23 47110ffebbeSMauro Carvalho Chehab 7-th fault Y: res=-1/23 47210ffebbeSMauro Carvalho Chehab 8-th fault Y: res=-1/12 47310ffebbeSMauro Carvalho Chehab 9-th fault Y: res=-1/12 47410ffebbeSMauro Carvalho Chehab 10-th fault Y: res=-1/12 47510ffebbeSMauro Carvalho Chehab 11-th fault Y: res=-1/12 47610ffebbeSMauro Carvalho Chehab 12-th fault Y: res=-1/12 47710ffebbeSMauro Carvalho Chehab 13-th fault Y: res=-1/12 47810ffebbeSMauro Carvalho Chehab 14-th fault Y: res=-1/12 47910ffebbeSMauro Carvalho Chehab 15-th fault Y: res=-1/12 48010ffebbeSMauro Carvalho Chehab 16-th fault N: res=0/12 481