1cfeb7489SZachary Loafman.\" 2cfeb7489SZachary Loafman.\" Copyright (c) 2009 Isilon Inc http://www.isilon.com/ 3cfeb7489SZachary Loafman.\" 4cfeb7489SZachary Loafman.\" Redistribution and use in source and binary forms, with or without 5cfeb7489SZachary Loafman.\" modification, are permitted provided that the following conditions 6cfeb7489SZachary Loafman.\" are met: 7cfeb7489SZachary Loafman.\" 1. Redistributions of source code must retain the above copyright 8cfeb7489SZachary Loafman.\" notice(s), this list of conditions and the following disclaimer as 9cfeb7489SZachary Loafman.\" the first lines of this file unmodified other than the possible 10cfeb7489SZachary Loafman.\" addition of one or more copyright notices. 11cfeb7489SZachary Loafman.\" 2. Redistributions in binary form must reproduce the above copyright 12cfeb7489SZachary Loafman.\" notice(s), this list of conditions and the following disclaimer in the 13cfeb7489SZachary Loafman.\" documentation and/or other materials provided with the distribution. 14cfeb7489SZachary Loafman.\" 15cfeb7489SZachary Loafman.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY 16cfeb7489SZachary Loafman.\" EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 17cfeb7489SZachary Loafman.\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 18cfeb7489SZachary Loafman.\" DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY 19cfeb7489SZachary Loafman.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 20cfeb7489SZachary Loafman.\" (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 21cfeb7489SZachary Loafman.\" SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 22cfeb7489SZachary Loafman.\" CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 23cfeb7489SZachary Loafman.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 24cfeb7489SZachary Loafman.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH 25cfeb7489SZachary Loafman.\" DAMAGE. 26cfeb7489SZachary Loafman.\" 27cfeb7489SZachary Loafman.\" $FreeBSD$ 28cfeb7489SZachary Loafman.\" 29cfeb7489SZachary Loafman.Dd May 10, 2009 30cfeb7489SZachary Loafman.Dt FAIL 9 31cfeb7489SZachary Loafman.Os 32cfeb7489SZachary Loafman.Sh NAME 33cfeb7489SZachary Loafman.Nm KFAIL_POINT_CODE , 34cfeb7489SZachary Loafman.Nm KFAIL_POINT_RETURN , 35cfeb7489SZachary Loafman.Nm KFAIL_POINT_RETURN_VOID , 36cfeb7489SZachary Loafman.Nm KFAIL_POINT_ERROR , 37cfeb7489SZachary Loafman.Nm KFAIL_POINT_GOTO , 38cfeb7489SZachary Loafman.Nm fail_point , 39cfeb7489SZachary Loafman.Nm DEBUG_FP 40cfeb7489SZachary Loafman. 41cfeb7489SZachary Loafman.Nd fail points 42cfeb7489SZachary Loafman.Sh SYNOPSIS 43cfeb7489SZachary Loafman.In sys/fail.h 44cfeb7489SZachary Loafman.Fn KFAIL_POINT_CODE "parent" "name" "code" 45cfeb7489SZachary Loafman.Fn KFAIL_POINT_RETURN "parent" "name" 46cfeb7489SZachary Loafman.Fn KFAIL_POINT_RETURN_VOID "parent" "name" 47cfeb7489SZachary Loafman.Fn KFAIL_POINT_ERROR "parent" "name" "error_var" 48cfeb7489SZachary Loafman.Fn KFAIL_POINT_GOTO "parent" "name" "error_var" "label" 49cfeb7489SZachary Loafman.Sh DESCRIPTION 50cfeb7489SZachary LoafmanFail points are used to add code points where errors may be injected 51eae608efSZachary Loafmanin a user controlled fashion. 52eae608efSZachary LoafmanFail points provide a convenient wrapper around user-provided error 53eae608efSZachary Loafmaninjection code, providing a 54eae608efSZachary Loafman.Xr sysctl 9 55eae608efSZachary LoafmanMIB, and a parser for that MIB that describes how the error 56cfeb7489SZachary Loafmaninjection code should fire. 57cfeb7489SZachary Loafman.Pp 58cfeb7489SZachary LoafmanThe base fail point macro is 59cfeb7489SZachary Loafman.Fn KFAIL_POINT_CODE 60cfeb7489SZachary Loafmanwhere 61cfeb7489SZachary Loafman.Fa parent 62cfeb7489SZachary Loafmanis a sysctl tree (frequently 63cfeb7489SZachary Loafman.Sy DEBUG_FP 64cfeb7489SZachary Loafmanfor kernel fail points, but various subsystems may wish to provide 65cfeb7489SZachary Loafmantheir own fail point trees), and 66cfeb7489SZachary Loafman.Fa name 67cfeb7489SZachary Loafmanis the name of the MIB in that tree, and 68cfeb7489SZachary Loafman.Fa code 69eae608efSZachary Loafmanis the error injection code. 70eae608efSZachary LoafmanThe 71cfeb7489SZachary Loafman.Fa code 72cfeb7489SZachary Loafmanargument does not require braces, but it is considered good style to 73eae608efSZachary Loafmanuse braces for any multi-line code arguments. 74eae608efSZachary LoafmanInside the 75cfeb7489SZachary Loafman.Fa code 76cfeb7489SZachary Loafmanargument, the evaluation of 77cfeb7489SZachary Loafman.Sy RETURN_VALUE 78cfeb7489SZachary Loafmanis derived from the 79cfeb7489SZachary Loafman.Fn return 80eae608efSZachary Loafmanvalue set in the sysctl MIB. 81eae608efSZachary LoafmanSee 82cfeb7489SZachary Loafman.Sx SYSCTL SETTINGS 83cfeb7489SZachary Loafmanbelow. 84cfeb7489SZachary Loafman.Pp 85cfeb7489SZachary LoafmanThe remaining 86cfeb7489SZachary Loafman.Fn KFAIL_POINT_* 87cfeb7489SZachary Loafmanmacros are wrappers around common error injection paths: 88cfeb7489SZachary Loafman.Bl -tag -width 8 89cfeb7489SZachary Loafman.It Fn KFAIL_POINT_RETURN parent name 90cfeb7489SZachary Loafmanis the equivalent of 91cfeb7489SZachary Loafman.Sy KFAIL_POINT_CODE(..., return RETURN_VALUE) 92cfeb7489SZachary Loafman.It Fn KFAIL_POINT_RETURN_VOID parent name 93cfeb7489SZachary Loafmanis the equivalent of 94cfeb7489SZachary Loafman.Sy KFAIL_POINT_CODE(..., return) 95cfeb7489SZachary Loafman.It Fn KFAIL_POINT_ERROR parent name error_var 96cfeb7489SZachary Loafmanis the equivalent of 97cfeb7489SZachary Loafman.Sy KFAIL_POINT_CODE(..., error_var = RETURN_VALUE) 98cfeb7489SZachary Loafman.It Fn KFAIL_POINT_GOTO parent name error_var label 99cfeb7489SZachary Loafmanis the equivalent of 100cfeb7489SZachary Loafman.Sy KFAIL_POINT_CODE(..., 101cfeb7489SZachary Loafman { error_var = RETURN_VALUE; goto label;}) 102cfeb7489SZachary Loafman.El 103cfeb7489SZachary Loafman.Pp 104cfeb7489SZachary Loafman.Sh SYSCTL VARIABLES 105cfeb7489SZachary LoafmanThe 106cfeb7489SZachary Loafman.Fn KFAIL_POINT_* 107eae608efSZachary Loafmanmacros add sysctl MIBs where specified. 108eae608efSZachary LoafmanMany base kernel MIBs can be found in the 109cfeb7489SZachary Loafman.Sy debug.fail_point 110cfeb7489SZachary Loafmantree (referenced in code by 111cfeb7489SZachary Loafman.Sy DEBUG_FP 112cfeb7489SZachary Loafman). 113cfeb7489SZachary Loafman.Pp 114eae608efSZachary LoafmanThe sysctl variable may be set using the following grammar: 115cfeb7489SZachary Loafman.Pp 116cfeb7489SZachary Loafman <fail_point> :: 117cfeb7489SZachary Loafman <term> ( "->" <term> )* 118cfeb7489SZachary Loafman.Pp 119cfeb7489SZachary Loafman <term> :: 120cfeb7489SZachary Loafman ( (<float> "%") | (<integer> "*" ) )* 121cfeb7489SZachary Loafman <type> 122cfeb7489SZachary Loafman [ "(" <integer> ")" ] 123cfeb7489SZachary Loafman.Pp 124cfeb7489SZachary Loafman <float> :: 125cfeb7489SZachary Loafman <integer> [ "." <integer> ] | 126cfeb7489SZachary Loafman "." <integer> 127cfeb7489SZachary Loafman.Pp 128cfeb7489SZachary Loafman <type> :: 129cfeb7489SZachary Loafman "off" | "return" | "sleep" | "panic" | "break" | "print" 130cfeb7489SZachary Loafman.Pp 131cfeb7489SZachary LoafmanThe <type> 132cfeb7489SZachary Loafmanargument specifies which action to take: 133cfeb7489SZachary Loafman.Bl -tag -width ".Dv return" 134cfeb7489SZachary Loafman.It Sy off 135cfeb7489SZachary LoafmanTake no action (does not trigger fail point code) 136cfeb7489SZachary Loafman.It Sy return 137cfeb7489SZachary LoafmanTrigger fail point code with specified argument 138cfeb7489SZachary Loafman.It Sy sleep 139cfeb7489SZachary LoafmanSleep the specified number of milliseconds 140cfeb7489SZachary Loafman.It Sy panic 141cfeb7489SZachary LoafmanPanic 142cfeb7489SZachary Loafman.It Sy break 143eae608efSZachary LoafmanBreak into the debugger, or trap if there is no debugger support 144cfeb7489SZachary Loafman.It Sy print 145cfeb7489SZachary LoafmanPrint that the fail point executed 146cfeb7489SZachary Loafman.El 147cfeb7489SZachary Loafman.Pp 148cfeb7489SZachary LoafmanThe <float>% and <integer>* modifiers prior to <type> control when 149eae608efSZachary Loafman<type> is executed. 150eae608efSZachary LoafmanThe <float>% form (e.g. "1.2%") can be used to specify a 151eae608efSZachary Loafmanprobability that <type> will execute. 152eae608efSZachary LoafmanThe <integer>* form (e.g. "5*") can be used to specify the number of 153eae608efSZachary Loafmantimes <type> should be executed before this <term> is disabled. 154eae608efSZachary LoafmanOnly the last probability and the last count are used if multiple 155eae608efSZachary Loafmanare specified, i.e. "1.2%2%" is the same as "2%". 156eae608efSZachary LoafmanWhen both a probability and a count are specified, the probability 157eae608efSZachary Loafmanis evaluated before the count, i.e. "2%5*" means "2% of the time, 158eae608efSZachary Loafmanbut only 5 times total". 159cfeb7489SZachary Loafman.Pp 160eae608efSZachary LoafmanThe operator -> can be used to express cascading terms. 161eae608efSZachary LoafmanIf you specify <term1>-><term2>, it means that if <term1> doesn't 162eae608efSZachary Loafman'execute', <term2> is evaluated. 163eae608efSZachary LoafmanFor the purpose of this operator, the return() and print() operators 164eae608efSZachary Loafmanare the only types that cascade. 165eae608efSZachary LoafmanA return() term only cascades if the code executes, and a print() 166eae608efSZachary Loafmanterm only cascades when passed a non-zero argument. 167cfeb7489SZachary Loafman.Pp 168cfeb7489SZachary Loafman.Sh EXAMPLES 169cfeb7489SZachary Loafman.Bl -tag 170cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="2.1%return(5)" 171cfeb7489SZachary Loafman21/1000ths of the time, execute 172cfeb7489SZachary Loafman.Fa code 173cfeb7489SZachary Loafmanwith RETURN_VALUE set to 5. 174cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="2%return(5)->5%return(22)" 175eae608efSZachary Loafman2/100ths of the time, execute 176cfeb7489SZachary Loafman.Fa code 177eae608efSZachary Loafmanwith RETURN_VALUE set to 5. 178eae608efSZachary LoafmanIf that doesn't happen, 5% of the time execute 179cfeb7489SZachary Loafman.Fa code 180cfeb7489SZachary Loafmanwith RETURN_VALUE set to 22. 181cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="5*return(5)->0.1%return(22)" 182eae608efSZachary LoafmanFor 5 times, return 5. 183eae608efSZachary LoafmanAfter that, 1/1000th of the time, return 22. 184cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="0.1%5*return(5)" 185eae608efSZachary LoafmanReturn 5 for 1 in 1000 executions, but only 5 times total. 186cfeb7489SZachary Loafman.It Sy sysctl debug.fail_point.foobar="1%*sleep(50)" 187eae608efSZachary Loafman1/100th of the time, sleep 50ms. 188cfeb7489SZachary Loafman.El 189cfeb7489SZachary Loafman.Pp 190cfeb7489SZachary Loafman.Sh CAVEATS 191cfeb7489SZachary LoafmanIt's easy to shoot yourself in the foot by setting fail points too 192eae608efSZachary Loafmanaggressively or setting too many in combination. 193eae608efSZachary LoafmanFor example, forcing 194cfeb7489SZachary Loafman.Fn malloc 195cfeb7489SZachary Loafmanto fail consistently is potentially harmful to uptime. 196cfeb7489SZachary Loafman.Pp 197cfeb7489SZachary LoafmanThe 198cfeb7489SZachary Loafman.Fn sleep 199eae608efSZachary Loafmansysctl setting may not be appropriate in all situations. 200eae608efSZachary LoafmanCurrently, 201cfeb7489SZachary Loafman.Fn fail_point_eval 202cfeb7489SZachary Loafmandoes not verify whether the context is appropriate for calling 203cfeb7489SZachary Loafman.Fn msleep . 204cfeb7489SZachary Loafman.Pp 205cfeb7489SZachary Loafman.Sh AUTHORS 206cfeb7489SZachary Loafman.An -nosplit 207cfeb7489SZachary LoafmanThis manual page was written by 208cfeb7489SZachary Loafman.An Zach Loafman Aq zml@FreeBSD.org . 209