1cfeb7489SZachary Loafman.\" 2cfeb7489SZachary Loafman.\" Copyright (c) 2009 Isilon Inc http://www.isilon.com/ 3cfeb7489SZachary Loafman.\" 4cfeb7489SZachary Loafman.\" Redistribution and use in source and binary forms, with or without 5cfeb7489SZachary Loafman.\" modification, are permitted provided that the following conditions 6cfeb7489SZachary Loafman.\" are met: 7cfeb7489SZachary Loafman.\" 1. Redistributions of source code must retain the above copyright 8cfeb7489SZachary Loafman.\" notice(s), this list of conditions and the following disclaimer as 9cfeb7489SZachary Loafman.\" the first lines of this file unmodified other than the possible 10cfeb7489SZachary Loafman.\" addition of one or more copyright notices. 11cfeb7489SZachary Loafman.\" 2. Redistributions in binary form must reproduce the above copyright 12cfeb7489SZachary Loafman.\" notice(s), this list of conditions and the following disclaimer in the 13cfeb7489SZachary Loafman.\" documentation and/or other materials provided with the distribution. 14cfeb7489SZachary Loafman.\" 15cfeb7489SZachary Loafman.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY 16cfeb7489SZachary Loafman.\" EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 17cfeb7489SZachary Loafman.\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 18cfeb7489SZachary Loafman.\" DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY 19cfeb7489SZachary Loafman.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 20cfeb7489SZachary Loafman.\" (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 21cfeb7489SZachary Loafman.\" SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 22cfeb7489SZachary Loafman.\" CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 23cfeb7489SZachary Loafman.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 24cfeb7489SZachary Loafman.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH 25cfeb7489SZachary Loafman.\" DAMAGE. 26cfeb7489SZachary Loafman.\" 27cfeb7489SZachary Loafman.\" $FreeBSD$ 28cfeb7489SZachary Loafman.\" 29cfeb7489SZachary Loafman.Dd May 10, 2009 30cfeb7489SZachary Loafman.Dt FAIL 9 31cfeb7489SZachary Loafman.Os 32cfeb7489SZachary Loafman.Sh NAME 33cfeb7489SZachary Loafman.Nm KFAIL_POINT_CODE , 34cfeb7489SZachary Loafman.Nm KFAIL_POINT_RETURN , 35cfeb7489SZachary Loafman.Nm KFAIL_POINT_RETURN_VOID , 36cfeb7489SZachary Loafman.Nm KFAIL_POINT_ERROR , 37cfeb7489SZachary Loafman.Nm KFAIL_POINT_GOTO , 38cfeb7489SZachary Loafman.Nm fail_point , 39cfeb7489SZachary Loafman.Nm DEBUG_FP 40cfeb7489SZachary Loafman.Nd fail points 41cfeb7489SZachary Loafman.Sh SYNOPSIS 42cfeb7489SZachary Loafman.In sys/fail.h 43cfeb7489SZachary Loafman.Fn KFAIL_POINT_CODE "parent" "name" "code" 44cfeb7489SZachary Loafman.Fn KFAIL_POINT_RETURN "parent" "name" 45cfeb7489SZachary Loafman.Fn KFAIL_POINT_RETURN_VOID "parent" "name" 46cfeb7489SZachary Loafman.Fn KFAIL_POINT_ERROR "parent" "name" "error_var" 47cfeb7489SZachary Loafman.Fn KFAIL_POINT_GOTO "parent" "name" "error_var" "label" 48cfeb7489SZachary Loafman.Sh DESCRIPTION 49cfeb7489SZachary LoafmanFail points are used to add code points where errors may be injected 50eae608efSZachary Loafmanin a user controlled fashion. 51eae608efSZachary LoafmanFail points provide a convenient wrapper around user-provided error 52eae608efSZachary Loafmaninjection code, providing a 53eae608efSZachary Loafman.Xr sysctl 9 54eae608efSZachary LoafmanMIB, and a parser for that MIB that describes how the error 55cfeb7489SZachary Loafmaninjection code should fire. 56cfeb7489SZachary Loafman.Pp 57cfeb7489SZachary LoafmanThe base fail point macro is 58cfeb7489SZachary Loafman.Fn KFAIL_POINT_CODE 59cfeb7489SZachary Loafmanwhere 60cfeb7489SZachary Loafman.Fa parent 61cfeb7489SZachary Loafmanis a sysctl tree (frequently 62cfeb7489SZachary Loafman.Sy DEBUG_FP 63cfeb7489SZachary Loafmanfor kernel fail points, but various subsystems may wish to provide 64cfeb7489SZachary Loafmantheir own fail point trees), and 65cfeb7489SZachary Loafman.Fa name 66cfeb7489SZachary Loafmanis the name of the MIB in that tree, and 67cfeb7489SZachary Loafman.Fa code 68eae608efSZachary Loafmanis the error injection code. 69eae608efSZachary LoafmanThe 70cfeb7489SZachary Loafman.Fa code 71cfeb7489SZachary Loafmanargument does not require braces, but it is considered good style to 72eae608efSZachary Loafmanuse braces for any multi-line code arguments. 73eae608efSZachary LoafmanInside the 74cfeb7489SZachary Loafman.Fa code 75cfeb7489SZachary Loafmanargument, the evaluation of 76cfeb7489SZachary Loafman.Sy RETURN_VALUE 77cfeb7489SZachary Loafmanis derived from the 78cfeb7489SZachary Loafman.Fn return 79eae608efSZachary Loafmanvalue set in the sysctl MIB. 80eae608efSZachary LoafmanSee 81aa8775c6SChristian Brueffer.Sx SYSCTL VARIABLES 82cfeb7489SZachary Loafmanbelow. 83cfeb7489SZachary Loafman.Pp 84cfeb7489SZachary LoafmanThe remaining 85cfeb7489SZachary Loafman.Fn KFAIL_POINT_* 86cfeb7489SZachary Loafmanmacros are wrappers around common error injection paths: 87cfeb7489SZachary Loafman.Bl -tag -width 8 88cfeb7489SZachary Loafman.It Fn KFAIL_POINT_RETURN parent name 89cfeb7489SZachary Loafmanis the equivalent of 90cfeb7489SZachary Loafman.Sy KFAIL_POINT_CODE(..., return RETURN_VALUE) 91cfeb7489SZachary Loafman.It Fn KFAIL_POINT_RETURN_VOID parent name 92cfeb7489SZachary Loafmanis the equivalent of 93cfeb7489SZachary Loafman.Sy KFAIL_POINT_CODE(..., return) 94cfeb7489SZachary Loafman.It Fn KFAIL_POINT_ERROR parent name error_var 95cfeb7489SZachary Loafmanis the equivalent of 96cfeb7489SZachary Loafman.Sy KFAIL_POINT_CODE(..., error_var = RETURN_VALUE) 97cfeb7489SZachary Loafman.It Fn KFAIL_POINT_GOTO parent name error_var label 98cfeb7489SZachary Loafmanis the equivalent of 99cfeb7489SZachary Loafman.Sy KFAIL_POINT_CODE(..., 100cfeb7489SZachary Loafman { error_var = RETURN_VALUE; goto label;}) 101cfeb7489SZachary Loafman.El 102cfeb7489SZachary Loafman.Sh SYSCTL VARIABLES 103cfeb7489SZachary LoafmanThe 104cfeb7489SZachary Loafman.Fn KFAIL_POINT_* 105eae608efSZachary Loafmanmacros add sysctl MIBs where specified. 106eae608efSZachary LoafmanMany base kernel MIBs can be found in the 107cfeb7489SZachary Loafman.Sy debug.fail_point 108cfeb7489SZachary Loafmantree (referenced in code by 109aa8775c6SChristian Brueffer.Sy DEBUG_FP ) . 110cfeb7489SZachary Loafman.Pp 111eae608efSZachary LoafmanThe sysctl variable may be set using the following grammar: 112cfeb7489SZachary Loafman.Pp 113aa8775c6SChristian Brueffer.Bd -literal 114cfeb7489SZachary Loafman <fail_point> :: 115cfeb7489SZachary Loafman <term> ( "->" <term> )* 116aa8775c6SChristian Brueffer 117cfeb7489SZachary Loafman <term> :: 118cfeb7489SZachary Loafman ( (<float> "%") | (<integer> "*" ) )* 119cfeb7489SZachary Loafman <type> 120cfeb7489SZachary Loafman [ "(" <integer> ")" ] 121aa8775c6SChristian Brueffer 122cfeb7489SZachary Loafman <float> :: 123cfeb7489SZachary Loafman <integer> [ "." <integer> ] | 124cfeb7489SZachary Loafman "." <integer> 125aa8775c6SChristian Brueffer 126cfeb7489SZachary Loafman <type> :: 127cfeb7489SZachary Loafman "off" | "return" | "sleep" | "panic" | "break" | "print" 128aa8775c6SChristian Brueffer.Ed 129cfeb7489SZachary Loafman.Pp 130aa8775c6SChristian BruefferThe <type> argument specifies which action to take: 131cfeb7489SZachary Loafman.Bl -tag -width ".Dv return" 132cfeb7489SZachary Loafman.It Sy off 133cfeb7489SZachary LoafmanTake no action (does not trigger fail point code) 134cfeb7489SZachary Loafman.It Sy return 135cfeb7489SZachary LoafmanTrigger fail point code with specified argument 136cfeb7489SZachary Loafman.It Sy sleep 137cfeb7489SZachary LoafmanSleep the specified number of milliseconds 138cfeb7489SZachary Loafman.It Sy panic 139cfeb7489SZachary LoafmanPanic 140cfeb7489SZachary Loafman.It Sy break 141eae608efSZachary LoafmanBreak into the debugger, or trap if there is no debugger support 142cfeb7489SZachary Loafman.It Sy print 143cfeb7489SZachary LoafmanPrint that the fail point executed 144cfeb7489SZachary Loafman.El 145cfeb7489SZachary Loafman.Pp 146cfeb7489SZachary LoafmanThe <float>% and <integer>* modifiers prior to <type> control when 147eae608efSZachary Loafman<type> is executed. 148eae608efSZachary LoafmanThe <float>% form (e.g. "1.2%") can be used to specify a 149eae608efSZachary Loafmanprobability that <type> will execute. 150eae608efSZachary LoafmanThe <integer>* form (e.g. "5*") can be used to specify the number of 151eae608efSZachary Loafmantimes <type> should be executed before this <term> is disabled. 152eae608efSZachary LoafmanOnly the last probability and the last count are used if multiple 153eae608efSZachary Loafmanare specified, i.e. "1.2%2%" is the same as "2%". 154eae608efSZachary LoafmanWhen both a probability and a count are specified, the probability 155eae608efSZachary Loafmanis evaluated before the count, i.e. "2%5*" means "2% of the time, 156eae608efSZachary Loafmanbut only 5 times total". 157cfeb7489SZachary Loafman.Pp 158eae608efSZachary LoafmanThe operator -> can be used to express cascading terms. 159aa8775c6SChristian BruefferIf you specify <term1>-><term2>, it means that if <term1> does not 160aa8775c6SChristian Brueffer.Ql execute , 161aa8775c6SChristian Brueffer<term2> is evaluated. 162eae608efSZachary LoafmanFor the purpose of this operator, the return() and print() operators 163eae608efSZachary Loafmanare the only types that cascade. 164eae608efSZachary LoafmanA return() term only cascades if the code executes, and a print() 165eae608efSZachary Loafmanterm only cascades when passed a non-zero argument. 166cfeb7489SZachary Loafman.Sh EXAMPLES 167cfeb7489SZachary Loafman.Bl -tag 168cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="2.1%return(5)" 169cfeb7489SZachary Loafman21/1000ths of the time, execute 170cfeb7489SZachary Loafman.Fa code 171cfeb7489SZachary Loafmanwith RETURN_VALUE set to 5. 172cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="2%return(5)->5%return(22)" 173eae608efSZachary Loafman2/100ths of the time, execute 174cfeb7489SZachary Loafman.Fa code 175eae608efSZachary Loafmanwith RETURN_VALUE set to 5. 176aa8775c6SChristian BruefferIf that does not happen, 5% of the time execute 177cfeb7489SZachary Loafman.Fa code 178cfeb7489SZachary Loafmanwith RETURN_VALUE set to 22. 179cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="5*return(5)->0.1%return(22)" 180eae608efSZachary LoafmanFor 5 times, return 5. 181eae608efSZachary LoafmanAfter that, 1/1000th of the time, return 22. 182cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="0.1%5*return(5)" 183eae608efSZachary LoafmanReturn 5 for 1 in 1000 executions, but only 5 times total. 184cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="1%*sleep(50)" 185eae608efSZachary Loafman1/100th of the time, sleep 50ms. 186cfeb7489SZachary Loafman.El 187cfeb7489SZachary Loafman.Sh CAVEATS 188aa8775c6SChristian BruefferIt is easy to shoot yourself in the foot by setting fail points too 189eae608efSZachary Loafmanaggressively or setting too many in combination. 190eae608efSZachary LoafmanFor example, forcing 191cfeb7489SZachary Loafman.Fn malloc 192cfeb7489SZachary Loafmanto fail consistently is potentially harmful to uptime. 193cfeb7489SZachary Loafman.Pp 194cfeb7489SZachary LoafmanThe 195cfeb7489SZachary Loafman.Fn sleep 196eae608efSZachary Loafmansysctl setting may not be appropriate in all situations. 197eae608efSZachary LoafmanCurrently, 198cfeb7489SZachary Loafman.Fn fail_point_eval 199cfeb7489SZachary Loafmandoes not verify whether the context is appropriate for calling 200cfeb7489SZachary Loafman.Fn msleep . 201cfeb7489SZachary Loafman.Sh AUTHORS 202cfeb7489SZachary Loafman.An -nosplit 203cfeb7489SZachary LoafmanThis manual page was written by 204cfeb7489SZachary Loafman.An Zach Loafman Aq zml@FreeBSD.org . 205