1.\" 2.\" CDDL HEADER START 3.\" 4.\" The contents of this file are subject to the terms of the 5.\" Common Development and Distribution License (the "License"). 6.\" You may not use this file except in compliance with the License. 7.\" 8.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 9.\" or https://opensource.org/licenses/CDDL-1.0. 10.\" See the License for the specific language governing permissions 11.\" and limitations under the License. 12.\" 13.\" When distributing Covered Code, include this CDDL HEADER in each 14.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE. 15.\" If applicable, add the following below this CDDL HEADER, with the 16.\" fields enclosed by brackets "[]" replaced with your own identifying 17.\" information: Portions Copyright [yyyy] [name of copyright owner] 18.\" 19.\" CDDL HEADER END 20.\" 21.\" Copyright 2013 Darik Horn <dajhorn@vanadac.com>. All rights reserved. 22.\" Copyright (c) 2024, Klara Inc. 23.\" 24.\" lint-ok: WARNING: sections out of conventional order: Sh SYNOPSIS 25.\" 26.Dd April 4, 2024 27.Dt ZINJECT 8 28.Os 29. 30.Sh NAME 31.Nm zinject 32.Nd ZFS Fault Injector 33.Sh DESCRIPTION 34.Nm 35creates artificial problems in a ZFS pool by simulating data corruption 36or device failures. 37This program is dangerous. 38. 39.Sh SYNOPSIS 40.Bl -tag -width Ds 41.It Xo 42.Nm zinject 43.Xc 44List injection records. 45. 46.It Xo 47.Nm zinject 48.Fl b Ar objset : Ns Ar object : Ns Ar level : Ns Ar start : Ns Ar end 49.Op Fl f Ar frequency 50.Fl amu 51.Op pool 52.Xc 53Force an error into the pool at a bookmark. 54. 55.It Xo 56.Nm zinject 57.Fl c Ar id Ns | Ns Sy all 58.Xc 59Cancel injection records. 60. 61.It Xo 62.Nm zinject 63.Fl d Ar vdev 64.Fl A Sy degrade Ns | Ns Sy fault 65.Ar pool 66.Xc 67Force a vdev into the DEGRADED or FAULTED state. 68. 69.It Xo 70.Nm zinject 71.Fl d Ar vdev 72.Fl D Ar latency : Ns Ar lanes 73.Op Fl T Ar read|write 74.Ar pool 75.Xc 76Add an artificial delay to I/O requests on a particular 77device, such that the requests take a minimum of 78.Ar latency 79milliseconds to complete. 80Each delay has an associated number of 81.Ar lanes 82which defines the number of concurrent 83I/O requests that can be processed. 84.Pp 85For example, with a single lane delay of 10 ms 86.No (\& Ns Fl D Ar 10 : Ns Ar 1 ) , 87the device will only be able to service a single I/O request 88at a time with each request taking 10 ms to complete. 89So, if only a single request is submitted every 10 ms, the 90average latency will be 10 ms; but if more than one request 91is submitted every 10 ms, the average latency will be more 92than 10 ms. 93.Pp 94Similarly, if a delay of 10 ms is specified to have two 95lanes 96.No (\& Ns Fl D Ar 10 : Ns Ar 2 ) , 97then the device will be able to service 98two requests at a time, each with a minimum latency of 10 ms. 99So, if two requests are submitted every 10 ms, then 100the average latency will be 10 ms; but if more than two 101requests are submitted every 10 ms, the average latency 102will be more than 10 ms. 103.Pp 104Also note, these delays are additive. 105So two invocations of 106.Fl D Ar 10 : Ns Ar 1 107are roughly equivalent to a single invocation of 108.Fl D Ar 10 : Ns Ar 2 . 109This also means, that one can specify multiple 110lanes with differing target latencies. 111For example, an invocation of 112.Fl D Ar 10 : Ns Ar 1 113followed by 114.Fl D Ar 25 : Ns Ar 2 115will create 3 lanes on the device: one lane with a latency 116of 10 ms and two lanes with a 25 ms latency. 117. 118.It Xo 119.Nm zinject 120.Fl d Ar vdev 121.Op Fl e Ar device_error 122.Op Fl L Ar label_error 123.Op Fl T Ar failure 124.Op Fl f Ar frequency 125.Op Fl F 126.Ar pool 127.Xc 128Force a vdev error. 129. 130.It Xo 131.Nm zinject 132.Fl i Ar seconds 133.Ar pool 134.Xc 135Add an artificial delay during the future import of a pool. 136This injector is automatically cleared after the import is finished. 137. 138.It Xo 139.Nm zinject 140.Fl I 141.Op Fl s Ar seconds Ns | Ns Fl g Ar txgs 142.Ar pool 143.Xc 144Simulate a hardware failure that fails to honor a cache flush. 145. 146.It Xo 147.Nm zinject 148.Fl p Ar function 149.Ar pool 150.Xc 151Panic inside the specified function. 152. 153.It Xo 154.Nm zinject 155.Fl t Sy data 156.Fl C Ar dvas 157.Op Fl e Ar device_error 158.Op Fl f Ar frequency 159.Op Fl l Ar level 160.Op Fl r Ar range 161.Op Fl amq 162.Ar path 163.Xc 164Force an error into the contents of a file. 165. 166.It Xo 167.Nm zinject 168.Fl t Sy dnode 169.Fl C Ar dvas 170.Op Fl e Ar device_error 171.Op Fl f Ar frequency 172.Op Fl l Ar level 173.Op Fl amq 174.Ar path 175.Xc 176Force an error into the metadnode for a file or directory. 177. 178.It Xo 179.Nm zinject 180.Fl t Ar mos_type 181.Fl C Ar dvas 182.Op Fl e Ar device_error 183.Op Fl f Ar frequency 184.Op Fl l Ar level 185.Op Fl r Ar range 186.Op Fl amqu 187.Ar pool 188.Xc 189Force an error into the MOS of a pool. 190.El 191.Sh OPTIONS 192.Bl -tag -width "-C dvas" 193.It Fl a 194Flush the ARC before injection. 195.It Fl b Ar objset : Ns Ar object : Ns Ar level : Ns Ar start : Ns Ar end 196Force an error into the pool at this bookmark tuple. 197Each number is in hexadecimal, and only one block can be specified. 198.It Fl C Ar dvas 199Inject the given error only into specific DVAs. 200The mask should be specified as a list of 0-indexed DVAs separated by commas 201.No (e.g . Ar 0,2 Ns No ). 202This option is not applicable to logical data errors such as 203.Sy decompress 204and 205.Sy decrypt . 206.It Fl d Ar vdev 207A vdev specified by path or GUID. 208.It Fl e Ar device_error 209Specify 210.Bl -tag -compact -width "decompress" 211.It Sy checksum 212for an ECKSUM error, 213.It Sy decompress 214for a data decompression error, 215.It Sy decrypt 216for a data decryption error, 217.It Sy corrupt 218to flip a bit in the data after a read, 219.It Sy dtl 220for an ECHILD error, 221.It Sy io 222for an EIO error where reopening the device will succeed, 223.It Sy nxio 224for an ENXIO error where reopening the device will fail, or 225.It Sy noop 226to drop the IO without executing it, and return success. 227.El 228.Pp 229For EIO and ENXIO, the "failed" reads or writes still occur. 230The probe simply sets the error value reported by the I/O pipeline 231so it appears the read or write failed. 232Decryption errors only currently work with file data. 233.It Fl f Ar frequency 234Only inject errors a fraction of the time. 235Expressed as a real number percentage between 236.Sy 0.0001 237and 238.Sy 100 . 239.It Fl F 240Fail faster. 241Do fewer checks. 242.It Fl f Ar txgs 243Run for this many transaction groups before reporting failure. 244.It Fl h 245Print the usage message. 246.It Fl l Ar level 247Inject an error at a particular block level. 248The default is 249.Sy 0 . 250.It Fl L Ar label_error 251Set the label error region to one of 252.Sy nvlist , 253.Sy pad1 , 254.Sy pad2 , 255or 256.Sy uber . 257.It Fl m 258Automatically remount the underlying filesystem. 259.It Fl q 260Quiet mode. 261Only print the handler number added. 262.It Fl r Ar range 263Inject an error over a particular logical range of an object, which 264will be translated to the appropriate blkid range according to the 265object's properties. 266.It Fl s Ar seconds 267Run for this many seconds before reporting failure. 268.It Fl T Ar failure 269Set the failure type to one of 270.Sy all , 271.Sy ioctl , 272.Sy claim , 273.Sy free , 274.Sy read , 275or 276.Sy write . 277.It Fl t Ar mos_type 278Set this to 279.Bl -tag -compact -width "spacemap" 280.It Sy mos 281for any data in the MOS, 282.It Sy mosdir 283for an object directory, 284.It Sy config 285for the pool configuration, 286.It Sy bpobj 287for the block pointer list, 288.It Sy spacemap 289for the space map, 290.It Sy metaslab 291for the metaslab, or 292.It Sy errlog 293for the persistent error log. 294.El 295.It Fl u 296Unload the pool after injection. 297.El 298. 299.Sh ENVIRONMENT VARIABLES 300.Bl -tag -width "ZF" 301.It Ev ZFS_HOSTID 302Run 303.Nm 304in debug mode. 305.El 306. 307.Sh SEE ALSO 308.Xr zfs 8 , 309.Xr zpool 8 310