1.\" SPDX-License-Identifier: CDDL-1.0 2.\" 3.\" CDDL HEADER START 4.\" 5.\" The contents of this file are subject to the terms of the 6.\" Common Development and Distribution License (the "License"). 7.\" You may not use this file except in compliance with the License. 8.\" 9.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE 10.\" or https://opensource.org/licenses/CDDL-1.0. 11.\" See the License for the specific language governing permissions 12.\" and limitations under the License. 13.\" 14.\" When distributing Covered Code, include this CDDL HEADER in each 15.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE. 16.\" If applicable, add the following below this CDDL HEADER, with the 17.\" fields enclosed by brackets "[]" replaced with your own identifying 18.\" information: Portions Copyright [yyyy] [name of copyright owner] 19.\" 20.\" CDDL HEADER END 21.\" 22.\" Copyright 2013 Darik Horn <dajhorn@vanadac.com>. All rights reserved. 23.\" Copyright (c) 2024, 2025, Klara, Inc. 24.\" 25.\" lint-ok: WARNING: sections out of conventional order: Sh SYNOPSIS 26.\" 27.Dd January 14, 2025 28.Dt ZINJECT 8 29.Os 30. 31.Sh NAME 32.Nm zinject 33.Nd ZFS Fault Injector 34.Sh DESCRIPTION 35.Nm 36creates artificial problems in a ZFS pool by simulating data corruption 37or device failures. 38This program is dangerous. 39. 40.Sh SYNOPSIS 41.Bl -tag -width Ds 42.It Xo 43.Nm zinject 44.Xc 45List injection records. 46. 47.It Xo 48.Nm zinject 49.Fl b Ar objset : Ns Ar object : Ns Ar level : Ns Ar start : Ns Ar end 50.Op Fl f Ar frequency 51.Fl amu 52.Op pool 53.Xc 54Force an error into the pool at a bookmark. 55. 56.It Xo 57.Nm zinject 58.Fl c Ar id Ns | Ns Sy all 59.Xc 60Cancel injection records. 61. 62.It Xo 63.Nm zinject 64.Fl d Ar vdev 65.Fl A Sy degrade Ns | Ns Sy fault 66.Ar pool 67.Xc 68Force a vdev into the DEGRADED or FAULTED state. 69. 70.It Xo 71.Nm zinject 72.Fl d Ar vdev 73.Fl D Ar latency : Ns Ar lanes 74.Op Fl T Ar read|write 75.Ar pool 76.Xc 77Add an artificial delay to I/O requests on a particular 78device, such that the requests take a minimum of 79.Ar latency 80milliseconds to complete. 81Each delay has an associated number of 82.Ar lanes 83which defines the number of concurrent 84I/O requests that can be processed. 85.Pp 86For example, with a single lane delay of 10 ms 87.No (\& Ns Fl D Ar 10 : Ns Ar 1 ) , 88the device will only be able to service a single I/O request 89at a time with each request taking 10 ms to complete. 90So, if only a single request is submitted every 10 ms, the 91average latency will be 10 ms; but if more than one request 92is submitted every 10 ms, the average latency will be more 93than 10 ms. 94.Pp 95Similarly, if a delay of 10 ms is specified to have two 96lanes 97.No (\& Ns Fl D Ar 10 : Ns Ar 2 ) , 98then the device will be able to service 99two requests at a time, each with a minimum latency of 10 ms. 100So, if two requests are submitted every 10 ms, then 101the average latency will be 10 ms; but if more than two 102requests are submitted every 10 ms, the average latency 103will be more than 10 ms. 104.Pp 105Also note, these delays are additive. 106So two invocations of 107.Fl D Ar 10 : Ns Ar 1 108are roughly equivalent to a single invocation of 109.Fl D Ar 10 : Ns Ar 2 . 110This also means, that one can specify multiple 111lanes with differing target latencies. 112For example, an invocation of 113.Fl D Ar 10 : Ns Ar 1 114followed by 115.Fl D Ar 25 : Ns Ar 2 116will create 3 lanes on the device: one lane with a latency 117of 10 ms and two lanes with a 25 ms latency. 118. 119.It Xo 120.Nm zinject 121.Fl d Ar vdev 122.Op Fl e Ar device_error 123.Op Fl L Ar label_error 124.Op Fl T Ar failure 125.Op Fl f Ar frequency 126.Op Fl F 127.Ar pool 128.Xc 129Force a vdev error. 130. 131.It Xo 132.Nm zinject 133.Fl i Ar seconds 134.Ar pool 135.Xc 136Add an artificial delay during the future import of a pool. 137This injector is automatically cleared after the import is finished. 138. 139.It Xo 140.Nm zinject 141.Fl E Ar delay 142.Op Fl a 143.Op Fl m 144.Op Fl f Ar freq 145.Op Fl l Ar level 146.Op Fl r Ar range 147.Op Fl T Ar iotype 148.Op Fl t Ar type Ns | Ns Fl b Ar bookmark 149.Xc 150Inject pipeline ready stage delays for the given object or bookmark. 151The delay is specified in milliseconds. 152. 153.It Xo 154.Nm zinject 155.Fl I 156.Op Fl s Ar seconds Ns | Ns Fl g Ar txgs 157.Ar pool 158.Xc 159Simulate a hardware failure that fails to honor a cache flush. 160. 161.It Xo 162.Nm zinject 163.Fl p Ar function 164.Ar pool 165.Xc 166Panic inside the specified function. 167. 168.It Xo 169.Nm zinject 170.Fl t Sy data 171.Fl C Ar dvas 172.Op Fl e Ar device_error 173.Op Fl f Ar frequency 174.Op Fl l Ar level 175.Op Fl r Ar range 176.Op Fl amq 177.Ar path 178.Xc 179Force an error into the contents of a file. 180. 181.It Xo 182.Nm zinject 183.Fl t Sy dnode 184.Fl C Ar dvas 185.Op Fl e Ar device_error 186.Op Fl f Ar frequency 187.Op Fl l Ar level 188.Op Fl amq 189.Ar path 190.Xc 191Force an error into the metadnode for a file or directory. 192. 193.It Xo 194.Nm zinject 195.Fl t Ar mos_type 196.Fl C Ar dvas 197.Op Fl e Ar device_error 198.Op Fl f Ar frequency 199.Op Fl l Ar level 200.Op Fl r Ar range 201.Op Fl amqu 202.Ar pool 203.Xc 204Force an error into the MOS of a pool. 205.El 206.Sh OPTIONS 207.Bl -tag -width "-C dvas" 208.It Fl a 209Flush the ARC before injection. 210.It Fl b Ar objset : Ns Ar object : Ns Ar level : Ns Ar start : Ns Ar end 211Force an error into the pool at this bookmark tuple. 212Each number is in hexadecimal, and only one block can be specified. 213.It Fl C Ar dvas 214Inject the given error only into specific DVAs. 215The mask should be specified as a list of 0-indexed DVAs separated by commas 216.No (e.g . Ar 0,2 Ns No ). 217This option is not applicable to logical data errors such as 218.Sy decompress 219and 220.Sy decrypt . 221.It Fl d Ar vdev 222A vdev specified by path or GUID. 223.It Fl e Ar device_error 224Specify 225.Bl -tag -compact -width "decompress" 226.It Sy checksum 227for an ECKSUM error, 228.It Sy decompress 229for a data decompression error, 230.It Sy decrypt 231for a data decryption error, 232.It Sy corrupt 233to flip a bit in the data after a read, 234.It Sy dtl 235for an ECHILD error, 236.It Sy io 237for an EIO error where reopening the device will succeed, 238.It Sy nxio 239for an ENXIO error where reopening the device will fail, or 240.It Sy noop 241to drop the IO without executing it, and return success. 242.El 243.Pp 244For EIO and ENXIO, the "failed" reads or writes still occur. 245The probe simply sets the error value reported by the I/O pipeline 246so it appears the read or write failed. 247Decryption errors only currently work with file data. 248.It Fl f Ar frequency 249Only inject errors a fraction of the time. 250Expressed as a real number percentage between 251.Sy 0.0001 252and 253.Sy 100 . 254.It Fl F 255Fail faster. 256Do fewer checks. 257.It Fl f Ar txgs 258Run for this many transaction groups before reporting failure. 259.It Fl h 260Print the usage message. 261.It Fl l Ar level 262Inject an error at a particular block level. 263The default is 264.Sy 0 . 265.It Fl L Ar label_error 266Set the label error region to one of 267.Sy nvlist , 268.Sy pad1 , 269.Sy pad2 , 270or 271.Sy uber . 272.It Fl m 273Automatically remount the underlying filesystem. 274.It Fl q 275Quiet mode. 276Only print the handler number added. 277.It Fl r Ar range 278Inject an error over a particular logical range of an object, which 279will be translated to the appropriate blkid range according to the 280object's properties. 281.It Fl s Ar seconds 282Run for this many seconds before reporting failure. 283.It Fl T Ar type 284Inject the error into I/O of this type. 285.Bl -tag -compact -width "read, write, flush, claim, free" 286.It Sy read , Sy write , Sy flush , Sy claim , Sy free 287Fundamental I/O types 288.It Sy all 289All fundamental I/O types 290.It Sy probe 291Device probe I/O 292.El 293.It Fl t Ar mos_type 294Set this to 295.Bl -tag -compact -width "spacemap" 296.It Sy mos 297for any data in the MOS, 298.It Sy mosdir 299for an object directory, 300.It Sy config 301for the pool configuration, 302.It Sy bpobj 303for the block pointer list, 304.It Sy spacemap 305for the space map, 306.It Sy metaslab 307for the metaslab, or 308.It Sy errlog 309for the persistent error log. 310.El 311.It Fl u 312Unload the pool after injection. 313.El 314. 315.Sh ENVIRONMENT VARIABLES 316.Bl -tag -width "ZF" 317.It Ev ZFS_HOSTID 318Run 319.Nm 320in debug mode. 321.El 322. 323.Sh SEE ALSO 324.Xr zfs 8 , 325.Xr zpool 8 326