1.\" 2.\" CDDL HEADER START 3.\" 4.\" The contents of this file are subject to the terms of the 5.\" Common Development and Distribution License (the "License"). 6.\" You may not use this file except in compliance with the License. 7.\" 8.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 9.\" or https://opensource.org/licenses/CDDL-1.0. 10.\" See the License for the specific language governing permissions 11.\" and limitations under the License. 12.\" 13.\" When distributing Covered Code, include this CDDL HEADER in each 14.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE. 15.\" If applicable, add the following below this CDDL HEADER, with the 16.\" fields enclosed by brackets "[]" replaced with your own identifying 17.\" information: Portions Copyright [yyyy] [name of copyright owner] 18.\" 19.\" CDDL HEADER END 20.\" 21.\" Copyright 2013 Darik Horn <dajhorn@vanadac.com>. All rights reserved. 22.\" Copyright (c) 2024, Klara Inc. 23.\" 24.\" lint-ok: WARNING: sections out of conventional order: Sh SYNOPSIS 25.\" 26.Dd April 4, 2024 27.Dt ZINJECT 8 28.Os 29. 30.Sh NAME 31.Nm zinject 32.Nd ZFS Fault Injector 33.Sh DESCRIPTION 34.Nm 35creates artificial problems in a ZFS pool by simulating data corruption 36or device failures. 37This program is dangerous. 38. 39.Sh SYNOPSIS 40.Bl -tag -width Ds 41.It Xo 42.Nm zinject 43.Xc 44List injection records. 45. 46.It Xo 47.Nm zinject 48.Fl b Ar objset : Ns Ar object : Ns Ar level : Ns Ar start : Ns Ar end 49.Op Fl f Ar frequency 50.Fl amu 51.Op pool 52.Xc 53Force an error into the pool at a bookmark. 54. 55.It Xo 56.Nm zinject 57.Fl c Ar id Ns | Ns Sy all 58.Xc 59Cancel injection records. 60. 61.It Xo 62.Nm zinject 63.Fl d Ar vdev 64.Fl A Sy degrade Ns | Ns Sy fault 65.Ar pool 66.Xc 67Force a vdev into the DEGRADED or FAULTED state. 68. 69.It Xo 70.Nm zinject 71.Fl d Ar vdev 72.Fl D Ar latency : Ns Ar lanes 73.Op Fl T Ar read|write 74.Ar pool 75.Xc 76Add an artificial delay to I/O requests on a particular 77device, such that the requests take a minimum of 78.Ar latency 79milliseconds to complete. 80Each delay has an associated number of 81.Ar lanes 82which defines the number of concurrent 83I/O requests that can be processed. 84.Pp 85For example, with a single lane delay of 10 ms 86.No (\& Ns Fl D Ar 10 : Ns Ar 1 ) , 87the device will only be able to service a single I/O request 88at a time with each request taking 10 ms to complete. 89So, if only a single request is submitted every 10 ms, the 90average latency will be 10 ms; but if more than one request 91is submitted every 10 ms, the average latency will be more 92than 10 ms. 93.Pp 94Similarly, if a delay of 10 ms is specified to have two 95lanes 96.No (\& Ns Fl D Ar 10 : Ns Ar 2 ) , 97then the device will be able to service 98two requests at a time, each with a minimum latency of 10 ms. 99So, if two requests are submitted every 10 ms, then 100the average latency will be 10 ms; but if more than two 101requests are submitted every 10 ms, the average latency 102will be more than 10 ms. 103.Pp 104Also note, these delays are additive. 105So two invocations of 106.Fl D Ar 10 : Ns Ar 1 107are roughly equivalent to a single invocation of 108.Fl D Ar 10 : Ns Ar 2 . 109This also means, that one can specify multiple 110lanes with differing target latencies. 111For example, an invocation of 112.Fl D Ar 10 : Ns Ar 1 113followed by 114.Fl D Ar 25 : Ns Ar 2 115will create 3 lanes on the device: one lane with a latency 116of 10 ms and two lanes with a 25 ms latency. 117. 118.It Xo 119.Nm zinject 120.Fl d Ar vdev 121.Op Fl e Ar device_error 122.Op Fl L Ar label_error 123.Op Fl T Ar failure 124.Op Fl f Ar frequency 125.Op Fl F 126.Ar pool 127.Xc 128Force a vdev error. 129. 130.It Xo 131.Nm zinject 132.Fl I 133.Op Fl s Ar seconds Ns | Ns Fl g Ar txgs 134.Ar pool 135.Xc 136Simulate a hardware failure that fails to honor a cache flush. 137. 138.It Xo 139.Nm zinject 140.Fl p Ar function 141.Ar pool 142.Xc 143Panic inside the specified function. 144. 145.It Xo 146.Nm zinject 147.Fl t Sy data 148.Fl C Ar dvas 149.Op Fl e Ar device_error 150.Op Fl f Ar frequency 151.Op Fl l Ar level 152.Op Fl r Ar range 153.Op Fl amq 154.Ar path 155.Xc 156Force an error into the contents of a file. 157. 158.It Xo 159.Nm zinject 160.Fl t Sy dnode 161.Fl C Ar dvas 162.Op Fl e Ar device_error 163.Op Fl f Ar frequency 164.Op Fl l Ar level 165.Op Fl amq 166.Ar path 167.Xc 168Force an error into the metadnode for a file or directory. 169. 170.It Xo 171.Nm zinject 172.Fl t Ar mos_type 173.Fl C Ar dvas 174.Op Fl e Ar device_error 175.Op Fl f Ar frequency 176.Op Fl l Ar level 177.Op Fl r Ar range 178.Op Fl amqu 179.Ar pool 180.Xc 181Force an error into the MOS of a pool. 182.El 183.Sh OPTIONS 184.Bl -tag -width "-C dvas" 185.It Fl a 186Flush the ARC before injection. 187.It Fl b Ar objset : Ns Ar object : Ns Ar level : Ns Ar start : Ns Ar end 188Force an error into the pool at this bookmark tuple. 189Each number is in hexadecimal, and only one block can be specified. 190.It Fl C Ar dvas 191Inject the given error only into specific DVAs. 192The mask should be specified as a list of 0-indexed DVAs separated by commas 193.No (e.g . Ar 0,2 Ns No ). 194This option is not applicable to logical data errors such as 195.Sy decompress 196and 197.Sy decrypt . 198.It Fl d Ar vdev 199A vdev specified by path or GUID. 200.It Fl e Ar device_error 201Specify 202.Bl -tag -compact -width "decompress" 203.It Sy checksum 204for an ECKSUM error, 205.It Sy decompress 206for a data decompression error, 207.It Sy decrypt 208for a data decryption error, 209.It Sy corrupt 210to flip a bit in the data after a read, 211.It Sy dtl 212for an ECHILD error, 213.It Sy io 214for an EIO error where reopening the device will succeed, 215.It Sy nxio 216for an ENXIO error where reopening the device will fail, or 217.It Sy noop 218to drop the IO without executing it, and return success. 219.El 220.Pp 221For EIO and ENXIO, the "failed" reads or writes still occur. 222The probe simply sets the error value reported by the I/O pipeline 223so it appears the read or write failed. 224Decryption errors only currently work with file data. 225.It Fl f Ar frequency 226Only inject errors a fraction of the time. 227Expressed as a real number percentage between 228.Sy 0.0001 229and 230.Sy 100 . 231.It Fl F 232Fail faster. 233Do fewer checks. 234.It Fl f Ar txgs 235Run for this many transaction groups before reporting failure. 236.It Fl h 237Print the usage message. 238.It Fl l Ar level 239Inject an error at a particular block level. 240The default is 241.Sy 0 . 242.It Fl L Ar label_error 243Set the label error region to one of 244.Sy nvlist , 245.Sy pad1 , 246.Sy pad2 , 247or 248.Sy uber . 249.It Fl m 250Automatically remount the underlying filesystem. 251.It Fl q 252Quiet mode. 253Only print the handler number added. 254.It Fl r Ar range 255Inject an error over a particular logical range of an object, which 256will be translated to the appropriate blkid range according to the 257object's properties. 258.It Fl s Ar seconds 259Run for this many seconds before reporting failure. 260.It Fl T Ar failure 261Set the failure type to one of 262.Sy all , 263.Sy ioctl , 264.Sy claim , 265.Sy free , 266.Sy read , 267or 268.Sy write . 269.It Fl t Ar mos_type 270Set this to 271.Bl -tag -compact -width "spacemap" 272.It Sy mos 273for any data in the MOS, 274.It Sy mosdir 275for an object directory, 276.It Sy config 277for the pool configuration, 278.It Sy bpobj 279for the block pointer list, 280.It Sy spacemap 281for the space map, 282.It Sy metaslab 283for the metaslab, or 284.It Sy errlog 285for the persistent error log. 286.El 287.It Fl u 288Unload the pool after injection. 289.El 290. 291.Sh ENVIRONMENT VARIABLES 292.Bl -tag -width "ZF" 293.It Ev ZFS_HOSTID 294Run 295.Nm 296in debug mode. 297.El 298. 299.Sh SEE ALSO 300.Xr zfs 8 , 301.Xr zpool 8 302