1=========================================== 2Fault injection capabilities infrastructure 3=========================================== 4 5See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug. 6 7 8Available fault injection capabilities 9-------------------------------------- 10 11- failslab 12 13 injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...) 14 15- fail_page_alloc 16 17 injects page allocation failures. (alloc_pages(), get_free_pages(), ...) 18 19- fail_usercopy 20 21 injects failures in user memory access functions. (copy_from_user(), get_user(), ...) 22 23- fail_futex 24 25 injects futex deadlock and uaddr fault errors. 26 27- fail_sunrpc 28 29 injects kernel RPC client and server failures. 30 31- fail_make_request 32 33 injects disk IO errors on devices permitted by setting 34 /sys/block/<device>/make-it-fail or 35 /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct()) 36 37- fail_mmc_request 38 39 injects MMC data errors on devices permitted by setting 40 debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request 41 42- fail_function 43 44 injects error return on specific functions, which are marked by 45 ALLOW_ERROR_INJECTION() macro, by setting debugfs entries 46 under /sys/kernel/debug/fail_function. No boot option supported. 47 48- fail_skb_realloc 49 50 inject skb (socket buffer) reallocation events into the network path. The 51 primary goal is to identify and prevent issues related to pointer 52 mismanagement in the network subsystem. By forcing skb reallocation at 53 strategic points, this feature creates scenarios where existing pointers to 54 skb headers become invalid. 55 56 When the fault is injected and the reallocation is triggered, cached pointers 57 to skb headers and data no longer reference valid memory locations. This 58 deliberate invalidation helps expose code paths where proper pointer updating 59 is neglected after a reallocation event. 60 61 By creating these controlled fault scenarios, the system can catch instances 62 where stale pointers are used, potentially leading to memory corruption or 63 system instability. 64 65 To select the interface to act on, write the network name to 66 /sys/kernel/debug/fail_skb_realloc/devname. 67 If this field is left empty (which is the default value), skb reallocation 68 will be forced on all network interfaces. 69 70 The effectiveness of this fault detection is enhanced when KASAN is 71 enabled, as it helps identify invalid memory references and use-after-free 72 (UAF) issues. 73 74- NVMe fault injection 75 76 inject NVMe status code and retry flag on devices permitted by setting 77 debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default 78 status code is NVME_SC_INVALID_OPCODE with no retry. The status code and 79 retry flag can be set via the debugfs. 80 81- Null test block driver fault injection 82 83 inject IO timeouts by setting config items under 84 /sys/kernel/config/nullb/<disk>/timeout_inject, 85 inject requeue requests by setting config items under 86 /sys/kernel/config/nullb/<disk>/requeue_inject, and 87 inject init_hctx() errors by setting config items under 88 /sys/kernel/config/nullb/<disk>/init_hctx_fault_inject. 89 90Configure fault-injection capabilities behavior 91----------------------------------------------- 92 93debugfs entries 94^^^^^^^^^^^^^^^ 95 96fault-inject-debugfs kernel module provides some debugfs entries for runtime 97configuration of fault-injection capabilities. 98 99- /sys/kernel/debug/fail*/probability: 100 101 likelihood of failure injection, in percent. 102 103 Format: <percent> 104 105 Note that one-failure-per-hundred is a very high error rate 106 for some testcases. Consider setting probability=100 and configure 107 /sys/kernel/debug/fail*/interval for such testcases. 108 109- /sys/kernel/debug/fail*/interval: 110 111 specifies the interval between failures, for calls to 112 should_fail() that pass all the other tests. 113 114 Note that if you enable this, by setting interval>1, you will 115 probably want to set probability=100. 116 117- /sys/kernel/debug/fail*/times: 118 119 specifies how many times failures may happen at most. A value of -1 120 means "no limit". 121 122- /sys/kernel/debug/fail*/space: 123 124 specifies an initial resource "budget", decremented by "size" 125 on each call to should_fail(,size). Failure injection is 126 suppressed until "space" reaches zero. 127 128- /sys/kernel/debug/fail*/verbose 129 130 Format: { 0 | 1 | 2 } 131 132 specifies the verbosity of the messages when failure is 133 injected. '0' means no messages; '1' will print only a single 134 log line per failure; '2' will print a call trace too -- useful 135 to debug the problems revealed by fault injection. 136 137- /sys/kernel/debug/fail*/task-filter: 138 139 Format: { 'Y' | 'N' } 140 141 A value of 'N' disables filtering by process (default). 142 Any positive value limits failures to only processes indicated by 143 /proc/<pid>/make-it-fail==1. 144 145- /sys/kernel/debug/fail*/require-start, 146 /sys/kernel/debug/fail*/require-end, 147 /sys/kernel/debug/fail*/reject-start, 148 /sys/kernel/debug/fail*/reject-end: 149 150 specifies the range of virtual addresses tested during 151 stacktrace walking. Failure is injected only if some caller 152 in the walked stacktrace lies within the required range, and 153 none lies within the rejected range. 154 Default required range is [0,ULONG_MAX) (whole of virtual address space). 155 Default rejected range is [0,0). 156 157- /sys/kernel/debug/fail*/stacktrace-depth: 158 159 specifies the maximum stacktrace depth walked during search 160 for a caller within [require-start,require-end) OR 161 [reject-start,reject-end). 162 163- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem: 164 165 Format: { 'Y' | 'N' } 166 167 default is 'Y', setting it to 'N' will also inject failures into 168 highmem/user allocations (__GFP_HIGHMEM allocations). 169 170- /sys/kernel/debug/failslab/cache-filter 171 Format: { 'Y' | 'N' } 172 173 default is 'N', setting it to 'Y' will only inject failures when 174 objects are requests from certain caches. 175 176 Select the cache by writing '1' to /sys/kernel/slab/<cache>/failslab: 177 178- /sys/kernel/debug/failslab/ignore-gfp-wait: 179- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait: 180 181 Format: { 'Y' | 'N' } 182 183 default is 'Y', setting it to 'N' will also inject failures 184 into allocations that can sleep (__GFP_DIRECT_RECLAIM allocations). 185 186- /sys/kernel/debug/fail_page_alloc/min-order: 187 188 specifies the minimum page allocation order to be injected 189 failures. 190 191- /sys/kernel/debug/fail_futex/ignore-private: 192 193 Format: { 'Y' | 'N' } 194 195 default is 'N', setting it to 'Y' will disable failure injections 196 when dealing with private (address space) futexes. 197 198- /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect: 199 200 Format: { 'Y' | 'N' } 201 202 default is 'N', setting it to 'Y' will disable disconnect 203 injection on the RPC client. 204 205- /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect: 206 207 Format: { 'Y' | 'N' } 208 209 default is 'N', setting it to 'Y' will disable disconnect 210 injection on the RPC server. 211 212- /sys/kernel/debug/fail_sunrpc/ignore-cache-wait: 213 214 Format: { 'Y' | 'N' } 215 216 default is 'N', setting it to 'Y' will disable cache wait 217 injection on the RPC server. 218 219- /sys/kernel/debug/fail_function/inject: 220 221 Format: { 'function-name' | '!function-name' | '' } 222 223 specifies the target function of error injection by name. 224 If the function name leads '!' prefix, given function is 225 removed from injection list. If nothing specified ('') 226 injection list is cleared. 227 228- /sys/kernel/debug/fail_function/injectable: 229 230 (read only) shows error injectable functions and what type of 231 error values can be specified. The error type will be one of 232 below; 233 - NULL: retval must be 0. 234 - ERRNO: retval must be -1 to -MAX_ERRNO (-4096). 235 - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096). 236 237- /sys/kernel/debug/fail_function/<function-name>/retval: 238 239 specifies the "error" return value to inject to the given function. 240 This will be created when the user specifies a new injection entry. 241 Note that this file only accepts unsigned values. So, if you want to 242 use a negative errno, you better use 'printf' instead of 'echo', e.g.: 243 $ printf %#x -12 > retval 244 245- /sys/kernel/debug/fail_skb_realloc/devname: 246 247 Specifies the network interface on which to force SKB reallocation. If 248 left empty, SKB reallocation will be applied to all network interfaces. 249 250 Example usage:: 251 252 # Force skb reallocation on eth0 253 echo "eth0" > /sys/kernel/debug/fail_skb_realloc/devname 254 255 # Clear the selection and force skb reallocation on all interfaces 256 echo "" > /sys/kernel/debug/fail_skb_realloc/devname 257 258Boot option 259^^^^^^^^^^^ 260 261In order to inject faults while debugfs is not available (early boot time), 262use the boot option:: 263 264 failslab= 265 fail_page_alloc= 266 fail_usercopy= 267 fail_make_request= 268 fail_futex= 269 fail_skb_realloc= 270 mmc_core.fail_request=<interval>,<probability>,<space>,<times> 271 272proc entries 273^^^^^^^^^^^^ 274 275- /proc/<pid>/fail-nth, 276 /proc/self/task/<tid>/fail-nth: 277 278 Write to this file of integer N makes N-th call in the task fail. 279 Read from this file returns a integer value. A value of '0' indicates 280 that the fault setup with a previous write to this file was injected. 281 A positive integer N indicates that the fault wasn't yet injected. 282 Note that this file enables all types of faults (slab, futex, etc). 283 This setting takes precedence over all other generic debugfs settings 284 like probability, interval, times, etc. But per-capability settings 285 (e.g. fail_futex/ignore-private) take precedence over it. 286 287 This feature is intended for systematic testing of faults in a single 288 system call. See an example below. 289 290 291Error Injectable Functions 292-------------------------- 293 294This part is for the kernel developers considering to add a function to 295ALLOW_ERROR_INJECTION() macro. 296 297Requirements for the Error Injectable Functions 298^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 299 300Since the function-level error injection forcibly changes the code path 301and returns an error even if the input and conditions are proper, this can 302cause unexpected kernel crash if you allow error injection on the function 303which is NOT error injectable. Thus, you (and reviewers) must ensure; 304 305- The function returns an error code if it fails, and the callers must check 306 it correctly (need to recover from it). 307 308- The function does not execute any code which can change any state before 309 the first error return. The state includes global or local, or input 310 variable. For example, clear output address storage (e.g. `*ret = NULL`), 311 increments/decrements counter, set a flag, preempt/irq disable or get 312 a lock (if those are recovered before returning error, that will be OK.) 313 314The first requirement is important, and it will result in that the release 315(free objects) functions are usually harder to inject errors than allocate 316functions. If errors of such release functions are not correctly handled 317it will cause a memory leak easily (the caller will confuse that the object 318has been released or corrupted.) 319 320The second one is for the caller which expects the function should always 321does something. Thus if the function error injection skips whole of the 322function, the expectation is betrayed and causes an unexpected error. 323 324Type of the Error Injectable Functions 325^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 326 327Each error injectable functions will have the error type specified by the 328ALLOW_ERROR_INJECTION() macro. You have to choose it carefully if you add 329a new error injectable function. If the wrong error type is chosen, the 330kernel may crash because it may not be able to handle the error. 331There are 4 types of errors defined in include/asm-generic/error-injection.h 332 333EI_ETYPE_NULL 334 This function will return `NULL` if it fails. e.g. return an allocated 335 object address. 336 337EI_ETYPE_ERRNO 338 This function will return an `-errno` error code if it fails. e.g. return 339 -EINVAL if the input is wrong. This will include the functions which will 340 return an address which encodes `-errno` by ERR_PTR() macro. 341 342EI_ETYPE_ERRNO_NULL 343 This function will return an `-errno` or `NULL` if it fails. If the caller 344 of this function checks the return value with IS_ERR_OR_NULL() macro, this 345 type will be appropriate. 346 347EI_ETYPE_TRUE 348 This function will return `true` (non-zero positive value) if it fails. 349 350If you specifies a wrong type, for example, EI_TYPE_ERRNO for the function 351which returns an allocated object, it may cause a problem because the returned 352value is not an object address and the caller can not access to the address. 353 354 355How to add new fault injection capability 356----------------------------------------- 357 358- #include <linux/fault-inject.h> 359 360- define the fault attributes 361 362 DECLARE_FAULT_ATTR(name); 363 364 Please see the definition of struct fault_attr in fault-inject.h 365 for details. 366 367- provide a way to configure fault attributes 368 369- boot option 370 371 If you need to enable the fault injection capability from boot time, you can 372 provide boot option to configure it. There is a helper function for it: 373 374 setup_fault_attr(attr, str); 375 376- debugfs entries 377 378 failslab, fail_page_alloc, fail_usercopy, and fail_make_request use this way. 379 Helper functions: 380 381 fault_create_debugfs_attr(name, parent, attr); 382 383- module parameters 384 385 If the scope of the fault injection capability is limited to a 386 single kernel module, it is better to provide module parameters to 387 configure the fault attributes. 388 389- add a hook to insert failures 390 391 Upon should_fail() returning true, client code should inject a failure: 392 393 should_fail(attr, size); 394 395Application Examples 396-------------------- 397 398- Inject slab allocation failures into module init/exit code:: 399 400 #!/bin/bash 401 402 FAILTYPE=failslab 403 echo Y > /sys/kernel/debug/$FAILTYPE/task-filter 404 echo 10 > /sys/kernel/debug/$FAILTYPE/probability 405 echo 100 > /sys/kernel/debug/$FAILTYPE/interval 406 echo -1 > /sys/kernel/debug/$FAILTYPE/times 407 echo 0 > /sys/kernel/debug/$FAILTYPE/space 408 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 409 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 410 411 faulty_system() 412 { 413 bash -c "echo 1 > /proc/self/make-it-fail && exec $*" 414 } 415 416 if [ $# -eq 0 ] 417 then 418 echo "Usage: $0 modulename [ modulename ... ]" 419 exit 1 420 fi 421 422 for m in $* 423 do 424 echo inserting $m... 425 faulty_system modprobe $m 426 427 echo removing $m... 428 faulty_system modprobe -r $m 429 done 430 431------------------------------------------------------------------------------ 432 433- Inject page allocation failures only for a specific module:: 434 435 #!/bin/bash 436 437 FAILTYPE=fail_page_alloc 438 module=$1 439 440 if [ -z $module ] 441 then 442 echo "Usage: $0 <modulename>" 443 exit 1 444 fi 445 446 modprobe $module 447 448 if [ ! -d /sys/module/$module/sections ] 449 then 450 echo Module $module is not loaded 451 exit 1 452 fi 453 454 cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start 455 cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end 456 457 echo N > /sys/kernel/debug/$FAILTYPE/task-filter 458 echo 10 > /sys/kernel/debug/$FAILTYPE/probability 459 echo 100 > /sys/kernel/debug/$FAILTYPE/interval 460 echo -1 > /sys/kernel/debug/$FAILTYPE/times 461 echo 0 > /sys/kernel/debug/$FAILTYPE/space 462 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 463 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 464 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem 465 echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth 466 467 trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT 468 469 echo "Injecting errors into the module $module... (interrupt to stop)" 470 sleep 1000000 471 472------------------------------------------------------------------------------ 473 474- Inject open_ctree error while btrfs mount:: 475 476 #!/bin/bash 477 478 rm -f testfile.img 479 dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1 480 DEVICE=$(losetup --show -f testfile.img) 481 mkfs.btrfs -f $DEVICE 482 mkdir -p tmpmnt 483 484 FAILTYPE=fail_function 485 FAILFUNC=open_ctree 486 echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject 487 printf %#x -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval 488 echo N > /sys/kernel/debug/$FAILTYPE/task-filter 489 echo 100 > /sys/kernel/debug/$FAILTYPE/probability 490 echo 0 > /sys/kernel/debug/$FAILTYPE/interval 491 echo -1 > /sys/kernel/debug/$FAILTYPE/times 492 echo 0 > /sys/kernel/debug/$FAILTYPE/space 493 echo 1 > /sys/kernel/debug/$FAILTYPE/verbose 494 495 mount -t btrfs $DEVICE tmpmnt 496 if [ $? -ne 0 ] 497 then 498 echo "SUCCESS!" 499 else 500 echo "FAILED!" 501 umount tmpmnt 502 fi 503 504 echo > /sys/kernel/debug/$FAILTYPE/inject 505 506 rmdir tmpmnt 507 losetup -d $DEVICE 508 rm testfile.img 509 510------------------------------------------------------------------------------ 511 512- Inject only skbuff allocation failures :: 513 514 # mark skbuff_head_cache as faulty 515 echo 1 > /sys/kernel/slab/skbuff_head_cache/failslab 516 # Turn on cache filter (off by default) 517 echo 1 > /sys/kernel/debug/failslab/cache-filter 518 # Turn on fault injection 519 echo 1 > /sys/kernel/debug/failslab/times 520 echo 1 > /sys/kernel/debug/failslab/probability 521 522 523Tool to run command with failslab or fail_page_alloc 524---------------------------------------------------- 525In order to make it easier to accomplish the tasks mentioned above, we can use 526tools/testing/fault-injection/failcmd.sh. Please run a command 527"./tools/testing/fault-injection/failcmd.sh --help" for more information and 528see the following examples. 529 530Examples: 531 532Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab 533allocation failure:: 534 535 # ./tools/testing/fault-injection/failcmd.sh \ 536 -- make -C tools/testing/selftests/ run_tests 537 538Same as above except to specify 100 times failures at most instead of one time 539at most by default:: 540 541 # ./tools/testing/fault-injection/failcmd.sh --times=100 \ 542 -- make -C tools/testing/selftests/ run_tests 543 544Same as above except to inject page allocation failure instead of slab 545allocation failure:: 546 547 # env FAILCMD_TYPE=fail_page_alloc \ 548 ./tools/testing/fault-injection/failcmd.sh --times=100 \ 549 -- make -C tools/testing/selftests/ run_tests 550 551Systematic faults using fail-nth 552--------------------------------- 553 554The following code systematically faults 0-th, 1-st, 2-nd and so on 555capabilities in the socketpair() system call:: 556 557 #include <sys/types.h> 558 #include <sys/stat.h> 559 #include <sys/socket.h> 560 #include <sys/syscall.h> 561 #include <fcntl.h> 562 #include <unistd.h> 563 #include <string.h> 564 #include <stdlib.h> 565 #include <stdio.h> 566 #include <errno.h> 567 568 int main() 569 { 570 int i, err, res, fail_nth, fds[2]; 571 char buf[128]; 572 573 system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait"); 574 sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid)); 575 fail_nth = open(buf, O_RDWR); 576 for (i = 1;; i++) { 577 sprintf(buf, "%d", i); 578 write(fail_nth, buf, strlen(buf)); 579 res = socketpair(AF_LOCAL, SOCK_STREAM, 0, fds); 580 err = errno; 581 pread(fail_nth, buf, sizeof(buf), 0); 582 if (res == 0) { 583 close(fds[0]); 584 close(fds[1]); 585 } 586 printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y', 587 res, err); 588 if (atoi(buf)) 589 break; 590 } 591 return 0; 592 } 593 594An example output:: 595 596 1-th fault Y: res=-1/23 597 2-th fault Y: res=-1/23 598 3-th fault Y: res=-1/12 599 4-th fault Y: res=-1/12 600 5-th fault Y: res=-1/23 601 6-th fault Y: res=-1/23 602 7-th fault Y: res=-1/23 603 8-th fault Y: res=-1/12 604 9-th fault Y: res=-1/12 605 10-th fault Y: res=-1/12 606 11-th fault Y: res=-1/12 607 12-th fault Y: res=-1/12 608 13-th fault Y: res=-1/12 609 14-th fault Y: res=-1/12 610 15-th fault Y: res=-1/12 611 16-th fault N: res=0/12 612