1.\" 2.\" CDDL HEADER START 3.\" 4.\" The contents of this file are subject to the terms of the 5.\" Common Development and Distribution License (the "License"). 6.\" You may not use this file except in compliance with the License. 7.\" 8.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 9.\" or http://www.opensolaris.org/os/licensing. 10.\" See the License for the specific language governing permissions 11.\" and limitations under the License. 12.\" 13.\" When distributing Covered Code, include this CDDL HEADER in each 14.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE. 15.\" If applicable, add the following below this CDDL HEADER, with the 16.\" fields enclosed by brackets "[]" replaced with your own identifying 17.\" information: Portions Copyright [yyyy] [name of copyright owner] 18.\" 19.\" CDDL HEADER END 20.\" 21.\" Copyright 2013 Darik Horn <dajhorn@vanadac.com>. All rights reserved. 22.\" 23.\" lint-ok: WARNING: sections out of conventional order: Sh SYNOPSIS 24.\" 25.Dd May 26, 2021 26.Dt ZINJECT 8 27.Os 28. 29.Sh NAME 30.Nm zinject 31.Nd ZFS Fault Injector 32.Sh DESCRIPTION 33.Nm 34creates artificial problems in a ZFS pool by simulating data corruption 35or device failures. 36This program is dangerous. 37. 38.Sh SYNOPSIS 39.Bl -tag -width Ds 40.It Xo 41.Nm zinject 42.Xc 43List injection records. 44. 45.It Xo 46.Nm zinject 47.Fl b Ar objset : Ns Ar object : Ns Ar level : Ns Ar start : Ns Ar end 48.Op Fl f Ar frequency 49.Fl amu 50.Op pool 51.Xc 52Force an error into the pool at a bookmark. 53. 54.It Xo 55.Nm zinject 56.Fl c Ar id Ns | Ns Sy all 57.Xc 58Cancel injection records. 59. 60.It Xo 61.Nm zinject 62.Fl d Ar vdev 63.Fl A Sy degrade Ns | Ns Sy fault 64.Ar pool 65.Xc 66Force a vdev into the DEGRADED or FAULTED state. 67. 68.It Xo 69.Nm zinject 70.Fl d Ar vdev 71.Fl D Ar latency : Ns Ar lanes 72.Ar pool 73.Xc 74Add an artificial delay to IO requests on a particular 75device, such that the requests take a minimum of 76.Ar latency 77milliseconds to complete. 78Each delay has an associated number of 79.Ar lanes 80which defines the number of concurrent 81IO requests that can be processed. 82.Pp 83For example, with a single lane delay of 10 ms 84.No (\& Ns Fl D Ar 10 : Ns Ar 1 ) , 85the device will only be able to service a single IO request 86at a time with each request taking 10 ms to complete. 87So, if only a single request is submitted every 10 ms, the 88average latency will be 10 ms; but if more than one request 89is submitted every 10 ms, the average latency will be more 90than 10 ms. 91.Pp 92Similarly, if a delay of 10 ms is specified to have two 93lanes 94.No (\& Ns Fl D Ar 10 : Ns Ar 2 ) , 95then the device will be able to service 96two requests at a time, each with a minimum latency of 10 ms. 97So, if two requests are submitted every 10 ms, then 98the average latency will be 10 ms; but if more than two 99requests are submitted every 10 ms, the average latency 100will be more than 10 ms. 101.Pp 102Also note, these delays are additive. 103So two invocations of 104.Fl D Ar 10 : Ns Ar 1 105are roughly equivalent to a single invocation of 106.Fl D Ar 10 : Ns Ar 2 . 107This also means, that one can specify multiple 108lanes with differing target latencies. 109For example, an invocation of 110.Fl D Ar 10 : Ns Ar 1 111followed by 112.Fl D Ar 25 : Ns Ar 2 113will create 3 lanes on the device: one lane with a latency 114of 10 ms and two lanes with a 25 ms latency. 115. 116.It Xo 117.Nm zinject 118.Fl d Ar vdev 119.Op Fl e Ar device_error 120.Op Fl L Ar label_error 121.Op Fl T Ar failure 122.Op Fl f Ar frequency 123.Op Fl F 124.Ar pool 125.Xc 126Force a vdev error. 127. 128.It Xo 129.Nm zinject 130.Fl I 131.Op Fl s Ar seconds Ns | Ns Fl g Ar txgs 132.Ar pool 133.Xc 134Simulate a hardware failure that fails to honor a cache flush. 135. 136.It Xo 137.Nm zinject 138.Fl p Ar function 139.Ar pool 140.Xc 141Panic inside the specified function. 142. 143.It Xo 144.Nm zinject 145.Fl t Sy data 146.Fl C Ar dvas 147.Op Fl e Ar device_error 148.Op Fl f Ar frequency 149.Op Fl l Ar level 150.Op Fl r Ar range 151.Op Fl amq 152.Ar path 153.Xc 154Force an error into the contents of a file. 155. 156.It Xo 157.Nm zinject 158.Fl t Sy dnode 159.Fl C Ar dvas 160.Op Fl e Ar device_error 161.Op Fl f Ar frequency 162.Op Fl l Ar level 163.Op Fl amq 164.Ar path 165.Xc 166Force an error into the metadnode for a file or directory. 167. 168.It Xo 169.Nm zinject 170.Fl t Ar mos_type 171.Fl C Ar dvas 172.Op Fl e Ar device_error 173.Op Fl f Ar frequency 174.Op Fl l Ar level 175.Op Fl r Ar range 176.Op Fl amqu 177.Ar pool 178.Xc 179Force an error into the MOS of a pool. 180.El 181.Sh OPTIONS 182.Bl -tag -width "-C dvas" 183.It Fl a 184Flush the ARC before injection. 185.It Fl b Ar objset : Ns Ar object : Ns Ar level : Ns Ar start : Ns Ar end 186Force an error into the pool at this bookmark tuple. 187Each number is in hexadecimal, and only one block can be specified. 188.It Fl C Ar dvas 189Inject the given error only into specific DVAs. 190The mask should be specified as a list of 0-indexed DVAs separated by commas 191.No (ex. Ar 0,2 Ns No ). 192This option is not applicable to logical data errors such as 193.Sy decompress 194and 195.Sy decrypt . 196.It Fl d Ar vdev 197A vdev specified by path or GUID. 198.It Fl e Ar device_error 199Specify 200.Bl -tag -compact -width "decompress" 201.It Sy checksum 202for an ECKSUM error, 203.It Sy decompress 204for a data decompression error, 205.It Sy decrypt 206for a data decryption error, 207.It Sy corrupt 208to flip a bit in the data after a read, 209.It Sy dtl 210for an ECHILD error, 211.It Sy io 212for an EIO error where reopening the device will succeed, or 213.It Sy nxio 214for an ENXIO error where reopening the device will fail. 215.El 216.Pp 217For EIO and ENXIO, the "failed" reads or writes still occur. 218The probe simply sets the error value reported by the I/O pipeline 219so it appears the read or write failed. 220Decryption errors only currently work with file data. 221.It Fl f Ar frequency 222Only inject errors a fraction of the time. 223Expressed as a real number percentage between 224.Sy 0.0001 225and 226.Sy 100 . 227.It Fl F 228Fail faster. 229Do fewer checks. 230.It Fl f Ar txgs 231Run for this many transaction groups before reporting failure. 232.It Fl h 233Print the usage message. 234.It Fl l Ar level 235Inject an error at a particular block level. 236The default is 237.Sy 0 . 238.It Fl L Ar label_error 239Set the label error region to one of 240.Sy nvlist , 241.Sy pad1 , 242.Sy pad2 , 243or 244.Sy uber . 245.It Fl m 246Automatically remount the underlying filesystem. 247.It Fl q 248Quiet mode. 249Only print the handler number added. 250.It Fl r Ar range 251Inject an error over a particular logical range of an object, which 252will be translated to the appropriate blkid range according to the 253object's properties. 254.It Fl s Ar seconds 255Run for this many seconds before reporting failure. 256.It Fl T Ar failure 257Set the failure type to one of 258.Sy all , 259.Sy claim , 260.Sy free , 261.Sy read , 262or 263.Sy write . 264.It Fl t Ar mos_type 265Set this to 266.Bl -tag -compact -width "spacemap" 267.It Sy mos 268for any data in the MOS, 269.It Sy mosdir 270for an object directory, 271.It Sy config 272for the pool configuration, 273.It Sy bpobj 274for the block pointer list, 275.It Sy spacemap 276for the space map, 277.It Sy metaslab 278for the metaslab, or 279.It Sy errlog 280for the persistent error log. 281.El 282.It Fl u 283Unload the pool after injection. 284.El 285. 286.Sh ENVIRONMENT VARIABLES 287.Bl -tag -width "ZF" 288.It Ev ZFS_HOSTID 289Run 290.Nm 291in debug mode. 292.El 293. 294.Sh SEE ALSO 295.Xr zfs 8 , 296.Xr zpool 8 297