1cfeb7489SZachary Loafman.\" 2cfeb7489SZachary Loafman.\" Copyright (c) 2009 Isilon Inc http://www.isilon.com/ 3cfeb7489SZachary Loafman.\" 4cfeb7489SZachary Loafman.\" Redistribution and use in source and binary forms, with or without 5cfeb7489SZachary Loafman.\" modification, are permitted provided that the following conditions 6cfeb7489SZachary Loafman.\" are met: 7cfeb7489SZachary Loafman.\" 1. Redistributions of source code must retain the above copyright 8cfeb7489SZachary Loafman.\" notice(s), this list of conditions and the following disclaimer as 9cfeb7489SZachary Loafman.\" the first lines of this file unmodified other than the possible 10cfeb7489SZachary Loafman.\" addition of one or more copyright notices. 11cfeb7489SZachary Loafman.\" 2. Redistributions in binary form must reproduce the above copyright 12cfeb7489SZachary Loafman.\" notice(s), this list of conditions and the following disclaimer in the 13cfeb7489SZachary Loafman.\" documentation and/or other materials provided with the distribution. 14cfeb7489SZachary Loafman.\" 15cfeb7489SZachary Loafman.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY 16cfeb7489SZachary Loafman.\" EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 17cfeb7489SZachary Loafman.\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 18cfeb7489SZachary Loafman.\" DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY 19cfeb7489SZachary Loafman.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 20cfeb7489SZachary Loafman.\" (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 21cfeb7489SZachary Loafman.\" SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 22cfeb7489SZachary Loafman.\" CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 23cfeb7489SZachary Loafman.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 24cfeb7489SZachary Loafman.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH 25cfeb7489SZachary Loafman.\" DAMAGE. 26cfeb7489SZachary Loafman.\" 27cfeb7489SZachary Loafman.\" $FreeBSD$ 28cfeb7489SZachary Loafman.\" 29cfeb7489SZachary Loafman.Dd May 10, 2009 30cfeb7489SZachary Loafman.Dt FAIL 9 31cfeb7489SZachary Loafman.Os 32cfeb7489SZachary Loafman.Sh NAME 33cfeb7489SZachary Loafman.Nm KFAIL_POINT_CODE , 34cfeb7489SZachary Loafman.Nm KFAIL_POINT_RETURN , 35cfeb7489SZachary Loafman.Nm KFAIL_POINT_RETURN_VOID , 36cfeb7489SZachary Loafman.Nm KFAIL_POINT_ERROR , 37cfeb7489SZachary Loafman.Nm KFAIL_POINT_GOTO , 38cfeb7489SZachary Loafman.Nm fail_point , 39cfeb7489SZachary Loafman.Nm DEBUG_FP 40cfeb7489SZachary Loafman. 41cfeb7489SZachary Loafman.Nd fail points 42cfeb7489SZachary Loafman.Sh SYNOPSIS 43cfeb7489SZachary Loafman.In sys/fail.h 44cfeb7489SZachary Loafman.Fn KFAIL_POINT_CODE "parent" "name" "code" 45cfeb7489SZachary Loafman.Fn KFAIL_POINT_RETURN "parent" "name" 46cfeb7489SZachary Loafman.Fn KFAIL_POINT_RETURN_VOID "parent" "name" 47cfeb7489SZachary Loafman.Fn KFAIL_POINT_ERROR "parent" "name" "error_var" 48cfeb7489SZachary Loafman.Fn KFAIL_POINT_GOTO "parent" "name" "error_var" "label" 49cfeb7489SZachary Loafman.Sh DESCRIPTION 50cfeb7489SZachary LoafmanFail points are used to add code points where errors may be injected 51cfeb7489SZachary Loafmanin a user controlled fashion. Fail points provide a convenient wrapper 52cfeb7489SZachary Loafmanaround user provided error injection code, providing a 53cfeb7489SZachary Loafman.Xr sysctl 9 MIB , and a parser for that MIB that describes how the error 54cfeb7489SZachary Loafmaninjection code should fire. 55cfeb7489SZachary Loafman.Pp 56cfeb7489SZachary LoafmanThe base fail point macro is 57cfeb7489SZachary Loafman.Fn KFAIL_POINT_CODE 58cfeb7489SZachary Loafmanwhere 59cfeb7489SZachary Loafman.Fa parent 60cfeb7489SZachary Loafmanis a sysctl tree (frequently 61cfeb7489SZachary Loafman.Sy DEBUG_FP 62cfeb7489SZachary Loafmanfor kernel fail points, but various subsystems may wish to provide 63cfeb7489SZachary Loafmantheir own fail point trees), and 64cfeb7489SZachary Loafman.Fa name 65cfeb7489SZachary Loafmanis the name of the MIB in that tree, and 66cfeb7489SZachary Loafman.Fa code 67cfeb7489SZachary Loafmanis the error injection code. The 68cfeb7489SZachary Loafman.Fa code 69cfeb7489SZachary Loafmanargument does not require braces, but it is considered good style to 70cfeb7489SZachary Loafmanuse braces for any multi-line code arguments. Inside the 71cfeb7489SZachary Loafman.Fa code 72cfeb7489SZachary Loafmanargument, the evaluation of 73cfeb7489SZachary Loafman.Sy RETURN_VALUE 74cfeb7489SZachary Loafmanis derived from the 75cfeb7489SZachary Loafman.Fn return 76cfeb7489SZachary Loafmanvalue set in the sysctl MIB. See 77cfeb7489SZachary Loafman.Sx SYSCTL SETTINGS 78cfeb7489SZachary Loafmanbelow. 79cfeb7489SZachary Loafman.Pp 80cfeb7489SZachary LoafmanThe remaining 81cfeb7489SZachary Loafman.Fn KFAIL_POINT_* 82cfeb7489SZachary Loafmanmacros are wrappers around common error injection paths: 83cfeb7489SZachary Loafman.Bl -tag -width 8 84cfeb7489SZachary Loafman.It Fn KFAIL_POINT_RETURN parent name 85cfeb7489SZachary Loafmanis the equivalent of 86cfeb7489SZachary Loafman.Sy KFAIL_POINT_CODE(..., return RETURN_VALUE) 87cfeb7489SZachary Loafman.It Fn KFAIL_POINT_RETURN_VOID parent name 88cfeb7489SZachary Loafmanis the equivalent of 89cfeb7489SZachary Loafman.Sy KFAIL_POINT_CODE(..., return) 90cfeb7489SZachary Loafman.It Fn KFAIL_POINT_ERROR parent name error_var 91cfeb7489SZachary Loafmanis the equivalent of 92cfeb7489SZachary Loafman.Sy KFAIL_POINT_CODE(..., error_var = RETURN_VALUE) 93cfeb7489SZachary Loafman.It Fn KFAIL_POINT_GOTO parent name error_var label 94cfeb7489SZachary Loafmanis the equivalent of 95cfeb7489SZachary Loafman.Sy KFAIL_POINT_CODE(..., 96cfeb7489SZachary Loafman { error_var = RETURN_VALUE; goto label;}) 97cfeb7489SZachary Loafman.El 98cfeb7489SZachary Loafman.Pp 99cfeb7489SZachary Loafman.Sh SYSCTL VARIABLES 100cfeb7489SZachary LoafmanThe 101cfeb7489SZachary Loafman.Fn KFAIL_POINT_* 102cfeb7489SZachary Loafmanmacros add sysctl MIBs where specified. Many base kernel MIBs can be 103cfeb7489SZachary Loafmanfound in the 104cfeb7489SZachary Loafman.Sy debug.fail_point 105cfeb7489SZachary Loafmantree (referenced in code by 106cfeb7489SZachary Loafman.Sy DEBUG_FP 107cfeb7489SZachary Loafman). 108cfeb7489SZachary Loafman.Pp 109cfeb7489SZachary LoafmanThe sysctl setting recognizes the following grammar: 110cfeb7489SZachary Loafman.Pp 111cfeb7489SZachary Loafman <fail_point> :: 112cfeb7489SZachary Loafman <term> ( "->" <term> )* 113cfeb7489SZachary Loafman.Pp 114cfeb7489SZachary Loafman <term> :: 115cfeb7489SZachary Loafman ( (<float> "%") | (<integer> "*" ) )* 116cfeb7489SZachary Loafman <type> 117cfeb7489SZachary Loafman [ "(" <integer> ")" ] 118cfeb7489SZachary Loafman.Pp 119cfeb7489SZachary Loafman <float> :: 120cfeb7489SZachary Loafman <integer> [ "." <integer> ] | 121cfeb7489SZachary Loafman "." <integer> 122cfeb7489SZachary Loafman.Pp 123cfeb7489SZachary Loafman <type> :: 124cfeb7489SZachary Loafman "off" | "return" | "sleep" | "panic" | "break" | "print" 125cfeb7489SZachary Loafman.Pp 126cfeb7489SZachary LoafmanThe <type> 127cfeb7489SZachary Loafmanargument specifies which action to take: 128cfeb7489SZachary Loafman.Bl -tag -width ".Dv return" 129cfeb7489SZachary Loafman.It Sy off 130cfeb7489SZachary LoafmanTake no action (does not trigger fail point code) 131cfeb7489SZachary Loafman.It Sy return 132cfeb7489SZachary LoafmanTrigger fail point code with specified argument 133cfeb7489SZachary Loafman.It Sy sleep 134cfeb7489SZachary LoafmanSleep the specified number of milliseconds 135cfeb7489SZachary Loafman.It Sy panic 136cfeb7489SZachary LoafmanPanic 137cfeb7489SZachary Loafman.It Sy break 138cfeb7489SZachary LoafmanBreak into the debugger. 139cfeb7489SZachary Loafman.It Sy print 140cfeb7489SZachary LoafmanPrint that the fail point executed 141cfeb7489SZachary Loafman.El 142cfeb7489SZachary Loafman.Pp 143cfeb7489SZachary LoafmanThe <float>% and <integer>* modifiers prior to <type> control when 144cfeb7489SZachary Loafman<type> is executed. The <float>% form (e.g. "1.2%") can be used to 145cfeb7489SZachary Loafmanspecify a probability that <type> will execute. The <integer>* form 146cfeb7489SZachary Loafman(e.g. "5*") can be used to specify the number of times <type> should 147cfeb7489SZachary Loafmanbe executed before this <term> is disabled. Only the last probability 148cfeb7489SZachary Loafmanand the last count are used if multiple are specified, i.e. "1.2%2%" 149cfeb7489SZachary Loafmanis the same as "2%". When both a probability and a count are 150cfeb7489SZachary Loafmanspecified, the probability is evaluated before the count, i.e. "2%5*" 151cfeb7489SZachary Loafmanmeans "2% of the time, but only execute it 5 times total". 152cfeb7489SZachary Loafman.Pp 153cfeb7489SZachary LoafmanThe operator -> can be used to express cascading terms. If you specify 154cfeb7489SZachary Loafman<term1>-><term2>, it means that if <term1> doesn't 'execute', <term2> 155cfeb7489SZachary Loafmanis evaluated. For the purpose of this operator, the return() and 156cfeb7489SZachary Loafmanprint() operators are the only types that cascade. A return() term 157cfeb7489SZachary Loafmanonly cascades if the code executes, and a print() term only cascades 158cfeb7489SZachary Loafmanwhen passed a non-zero argument. 159cfeb7489SZachary Loafman.Pp 160cfeb7489SZachary Loafman.Sh EXAMPLES 161cfeb7489SZachary Loafman.Bl -tag 162cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="2.1%return(5)" 163cfeb7489SZachary Loafman21/1000ths of the time, execute 164cfeb7489SZachary Loafman.Fa code 165cfeb7489SZachary Loafmanwith RETURN_VALUE set to 5. 166cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="2%return(5)->5%return(22)" 167cfeb7489SZachary Loafman2/100th of the time, execute 168cfeb7489SZachary Loafman.Fa code 169cfeb7489SZachary Loafmanwith RETURN_VALUE set to 5. If that doesn't happen, 5% of the time 170cfeb7489SZachary Loafmanexecute 171cfeb7489SZachary Loafman.Fa code 172cfeb7489SZachary Loafmanwith RETURN_VALUE set to 22. 173cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="5*return(5)->0.1%return(22)" 174cfeb7489SZachary LoafmanFor 5 times, return 5. After that, 1/1000ths of the time, return 22. 175cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="0.1%5*return(5)" 176cfeb7489SZachary LoafmanReturn 5 for 1 in 1000 executions, but only execute 5 times total. 177cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="1%*sleep(50)" 178cfeb7489SZachary Loafman1/100ths of the time, sleep 50ms. 179cfeb7489SZachary Loafman.El 180cfeb7489SZachary Loafman.Pp 181cfeb7489SZachary Loafman.Sh CAVEATS 182cfeb7489SZachary LoafmanIt's easy to shoot yourself in the foot by setting fail points too 183cfeb7489SZachary Loafmanaggressively or setting too many in combination. For example, forcing 184cfeb7489SZachary Loafman.Fn malloc 185cfeb7489SZachary Loafmanto fail consistently is potentially harmful to uptime. 186cfeb7489SZachary Loafman.Pp 187cfeb7489SZachary LoafmanThe 188cfeb7489SZachary Loafman.Fn sleep 189cfeb7489SZachary Loafmansysctl setting may not be appropriate in all situations. Currently, 190cfeb7489SZachary Loafman.Fn fail_point_eval 191cfeb7489SZachary Loafmandoes not verify whether the context is appropriate for calling 192cfeb7489SZachary Loafman.Fn msleep . 193cfeb7489SZachary Loafman.Pp 194cfeb7489SZachary Loafman.Sh AUTHORS 195cfeb7489SZachary Loafman.An -nosplit 196cfeb7489SZachary LoafmanThis manual page was written by 197cfeb7489SZachary Loafman.An Zach Loafman Aq zml@FreeBSD.org . 198